-
Notifications
You must be signed in to change notification settings - Fork 2
/
Copy pathatom.xml
636 lines (304 loc) · 230 KB
/
atom.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
<title>亂馬客</title>
<icon>https://www.gravatar.com/avatar/cd3aed042ccd7a5a5d9956b0bc07dc81</icon>
<subtitle>Hello! 我是RM</subtitle>
<link href="https://rainmakerho.github.io/atom.xml" rel="self"/>
<link href="https://rainmakerho.github.io/"/>
<updated>2025-04-12T06:30:26.858Z</updated>
<id>https://rainmakerho.github.io/</id>
<author>
<name>亂馬客</name>
<email>[email protected]</email>
</author>
<generator uri="https://hexo.io/">Hexo</generator>
<entry>
<title>Dify Docker Compose SSL 憑證驗證失敗排查與修復指南（Ubuntu 環境實戰）</title>
<link href="https://rainmakerho.github.io/2025/04/12/ubuntu-dify-docker-ssl-cert-fix/"/>
<id>https://rainmakerho.github.io/2025/04/12/ubuntu-dify-docker-ssl-cert-fix/</id>
<published>2025-04-12T05:16:43.000Z</published>
<updated>2025-04-12T06:30:26.858Z</updated>
<content type="html"><![CDATA[<h3 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h3><p>今天原本使用好好的政府 OpenData API (<a href="https://gcis.nat.gov.tw/">https://gcis.nat.gov.tw</a>) 在 Dify 的 Http 節點中居然出現<code>[SSL:CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1010)</code>的錯誤。<br>以下紀錄在部署 Dify 至 Ubuntu Docker Compose 環境時，遇到 SSL 憑證驗證失敗的問題，包含台灣政府 CA 安裝、Python certifi 處理、Proxy 設定與完整排查修復流程。</p><h3 id="問題"><a href="#問題" class="headerlink" title="問題"></a>問題</h3><p>先在 Ubuntu 執行 curl 就出現<strong>curl: (60) SSL certificate problem: unable to get local issuer certificate</strong>的錯誤。</p><h5 id="Step-1-確認-ubuntu-SSL-問題"><a href="#Step-1-確認-ubuntu-SSL-問題" class="headerlink" title="Step 1.確認 ubuntu SSL 問題"></a>Step 1.確認 ubuntu SSL 問題</h5><p>使用 openssl 檢查憑證鏈：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">openssl s_client -connect data.gcis.nat.gov.tw:443</span><br></pre></td></tr></table></figure><p>結果如下，</p><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><span class="line">Server certificate</span><br><span class="line">subject=C = TW, L = \E8\87\BA\E5\8C\97\E5\B8\82, O = \E6\94\BF\E5\BA\9C\E6\A9\9F\E9\97\9C-\E8\B3\87\E8\A8\8A\E5\AE\A4, CN = gcis.nat.gov.tw</span><br><span class="line">issuer=C = TW, O = "Chunghwa Telecom Co., Ltd.", CN = HiPKI OV TLS CA - G1</span><br><span class="line">---</span><br><span class="line">No client certificate CA names sent</span><br><span class="line">Peer signing digest: SHA256</span><br><span class="line">Peer signature type: RSA-PSS</span><br><span class="line">Server Temp Key: ECDH, prime256v1, 256 bits</span><br><span class="line">---</span><br><span class="line">SSL handshake has read 7920 bytes and written 448 bytes</span><br><span class="line">Verification error: unable to verify the first certificate</span><br><span class="line">---</span><br><span class="line">New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384</span><br><span class="line">Server public key is 2048 bit</span><br><span class="line">Secure Renegotiation IS supported</span><br><span class="line">Compression: NONE</span><br><span class="line">Expansion: NONE</span><br></pre></td></tr></table></figure><p><strong>Verification error: unable to verify the first certificate</strong>可以發現沒有 Root CA。</p><h5 id="Step-2-安裝-HiPKI-OV-TLS-CA"><a href="#Step-2-安裝-HiPKI-OV-TLS-CA" class="headerlink" title="Step 2.安裝 HiPKI OV TLS CA"></a>Step 2.安裝 HiPKI OV TLS CA</h5><p>到<a href="https://epki.com.tw/repository-h/index.htm">中華電信公開憑證</a>下載<a href="https://epki.com.tw/repository-h/download/OVTLSCA1_b64.crt">HiPKI OV TLS CA</a>，<br>並放到 <code>/usr/local/share/ca-certificates/</code>目錄中</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">wget https://epki.com.tw/repository-h/download/OVTLSCA1_b64.crt -O /usr/local/share/ca-certificates/OVTLSCA1_b64.crt</span><br></pre></td></tr></table></figure><p>更新系統 CA</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">sudo update-ca-certificates</span><br></pre></td></tr></table></figure><p>結果如下，</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">1 added, 0 removed; <span class="keyword">done</span>.</span><br><span class="line">Running hooks <span class="keyword">in</span> /etc/ca-certificates/update.d...</span><br><span class="line"><span class="keyword">done</span>.</span><br></pre></td></tr></table></figure><p>再使用 openssl 檢查憑證鏈<code>echo | openssl s_client -showcerts -connect data.gcis.nat.gov.tw:443</code>可以發現已經驗證成功，</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">SSL handshake has <span class="built_in">read</span> 7920 bytes and written 448 bytes</span><br><span class="line">Verification: OK</span><br></pre></td></tr></table></figure><p>完成後，再到 Dify 執行，還是出現<code>[SSL:CERTIFICATE_VERIFY_FAILED] </code>錯誤，<br>接下來，要來查看 Dify Docker 中的問題</p><h5 id="Step-3-查看-Api-Log"><a href="#Step-3-查看-Api-Log" class="headerlink" title="Step 3.查看 Api Log"></a>Step 3.查看 Api Log</h5><p>查看 Api 的 Log</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">docker compose logs api</span><br></pre></td></tr></table></figure><p>輸出如下，</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">[ssrf_proxy.py:78] - Request to URL http://data.gcis.nat.gov.tw/od/data/api/5F64D864-61CB-4D0D-8AD9-492047CC1EA6?<span class="variable">$format</span>=json&<span class="variable">$filter</span>=Business_Accounting_NO eq 22425662 failed on attempt 1: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get <span class="built_in">local</span> issuer certificate (_ssl.c:1010)</span><br><span class="line">[node.py:103] - http request node 1743067744592 failed to run: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get <span class="built_in">local</span> issuer certificate (_ssl.c:1010)</span><br></pre></td></tr></table></figure><h5 id="Step-4-確認-API-Container-的憑證環境"><a href="#Step-4-確認-API-Container-的憑證環境" class="headerlink" title="Step 4.確認 API Container 的憑證環境"></a>Step 4.確認 API Container 的憑證環境</h5><p>進入 API Container</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">docker compose <span class="built_in">exec</span> api bash</span><br></pre></td></tr></table></figure><p>確認 Python certifi 使用的憑證：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">python3 -c <span class="string">"import certifi; print(certifi.where())"</span></span><br></pre></td></tr></table></figure><p>結果如下，</p><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">/app/api/.venv/lib/python3.12/site-packages/certifi/cacert.pem</span><br></pre></td></tr></table></figure><h5 id="Step-5-替換-certifi-的憑證檔"><a href="#Step-5-替換-certifi-的憑證檔" class="headerlink" title="Step 5.替換 certifi 的憑證檔"></a>Step 5.替換 certifi 的憑證檔</h5><p>ubuntu host 的憑證是 OK 的，所以可以在<code>docker-compose.yaml</code>設定 volume，<br>修改 dify 的 docker-compose.yaml 在 api 的 volumes 加入以下設定</p><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="attr">api:</span></span><br><span class="line"> <span class="attr">volumes:</span></span><br><span class="line"> <span class="bullet">-</span> <span class="string">/etc/ssl/certs/ca-certificates.crt:/etc/ssl/certs/ca-certificates.crt:ro</span></span><br><span class="line"> <span class="bullet">-</span> <span class="string">/etc/ssl/certs/ca-certificates.crt:/app/api/.venv/lib/python3.12/site-packages/certifi/cacert.pem:ro</span></span><br></pre></td></tr></table></figure><ul><li>將 VM Host 上最新的 CA cert 掛到 container 的 OS 層 /etc/ssl/certs/ca-certificates.crt</li><li>將 VM Host 的 CA cert 掛到 Python certifi 套件的 cert 檔案路徑</li></ul><p>重新執行<code>docker compose</code>並測試看看，</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">docker compose down</span><br><span class="line">docker compose up -d</span><br></pre></td></tr></table></figure><p>再從 dify 去測試，可以發現 Http 節點，已經可以順利執行成功了。</p><h3 id="結語"><a href="#結語" class="headerlink" title="結語"></a>結語</h3><p>以上的 SSL 問題排查流程，驗證了一個 Containerized 架構下的典型 SSL 問題：</p><ol><li>VM 憑證要先驗證成功。</li><li>Docker Container 須確認 CA cert 路徑。</li><li>Python 的 certifi 使用獨立 cert，若寫死 verify=certifi.where() 就要特別處理。</li></ol><h3 id="參考資源"><a href="#參考資源" class="headerlink" title="參考資源"></a>參考資源</h3><p><a href="https://epki.com.tw/repository-h/index.htm">中華電信公開憑證</a><br><a href="https://requests.readthedocs.io/en/latest/user/advanced/#ssl-cert-verification">SSL Cert Verification</a><br><a href="https://documentation.ubuntu.com/server/how-to/security/install-a-root-ca-certificate-in-the-trust-store/index.html">Install a root CA certificate in the trust store</a><br><a href="https://manpages.ubuntu.com/manpages/focal/man8/update-ca-certificates.8.html">update-ca-certificates 說明</a></p>]]></content>
<summary type="html"><h3 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h3><p>今天原本使用好好的政府 OpenData API (<a href="https://gcis.nat.gov.tw/">https://g</summary>
<category term="ubuntu" scheme="https://rainmakerho.github.io/tags/ubuntu/"/>
<category term="docker" scheme="https://rainmakerho.github.io/tags/docker/"/>
<category term="dify" scheme="https://rainmakerho.github.io/tags/dify/"/>
<category term="ssl" scheme="https://rainmakerho.github.io/tags/ssl/"/>
<category term="certificate" scheme="https://rainmakerho.github.io/tags/certificate/"/>
<category term="python" scheme="https://rainmakerho.github.io/tags/python/"/>
<category term="certifi" scheme="https://rainmakerho.github.io/tags/certifi/"/>
<category term="docker-compose" scheme="https://rainmakerho.github.io/tags/docker-compose/"/>
<category term="台灣政府憑證" scheme="https://rainmakerho.github.io/tags/%E5%8F%B0%E7%81%A3%E6%94%BF%E5%BA%9C%E6%86%91%E8%AD%89/"/>
<category term="root ca" scheme="https://rainmakerho.github.io/tags/root-ca/"/>
<category term="gcis.nat.gov.tw" scheme="https://rainmakerho.github.io/tags/gcis-nat-gov-tw/"/>
<category term="憑證問題" scheme="https://rainmakerho.github.io/tags/%E6%86%91%E8%AD%89%E5%95%8F%E9%A1%8C/"/>
</entry>
<entry>
<title>使用 Kernel Memory 的 Decoders 來幫取得檔案、網頁..的文字</title>
<link href="https://rainmakerho.github.io/2025/03/27/kernel-memory-file-decoder/"/>
<id>https://rainmakerho.github.io/2025/03/27/kernel-memory-file-decoder/</id>
<published>2025-03-27T06:42:26.000Z</published>
<updated>2025-03-27T06:55:29.776Z</updated>
<content type="html"><![CDATA[<h3 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h3><p>在<a href="https://rainmakerho.github.io/2025/02/19/kernel-memory-plaintext-chunker/">使用 Kernel Memory 的 TextChunker 來幫我們切 Chunk</a>中將文字切段，<br>那裡怎麼取得檔案中的文字呢?<br>在<strong>Kernel Memory</strong>一樣有提供基本的<strong>Decoder</strong>來讓我們使用，<br>以下就來建立一個<code>ExtractFile</code>Method 來取出檔案的文字 …</p><h3 id="實作"><a href="#實作" class="headerlink" title="實作"></a>實作</h3><p>1.加入<code>Microsoft.KernelMemory</code> Nuget 套件</p><p>2.建立一個<code>ExtractFile</code>Method</p><figure class="highlight csharp"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#<span class="keyword">pragma</span> <span class="keyword">warning</span> disable KMEXP00</span></span><br><span class="line"><span class="keyword">private</span> <span class="keyword">static</span> ILoggerFactory loggerFactory;</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">static</span> <span class="keyword">async</span> Task<<span class="built_in">string</span>> <span class="title">ExtractFile</span>(<span class="params"><span class="built_in">string</span> docPath, <span class="built_in">bool</span> isUrl = <span class="literal">false</span></span>)</span></span><br><span class="line">{</span><br><span class="line"> <span class="keyword">var</span> mimeTypeDetection = <span class="keyword">new</span> MimeTypesDetection();</span><br><span class="line"> <span class="built_in">string</span> mimeType;</span><br><span class="line"> BinaryData? fileBinary;</span><br><span class="line"> <span class="keyword">if</span> (isUrl)</span><br><span class="line"> {</span><br><span class="line"> <span class="keyword">var</span> webscraper = <span class="keyword">new</span> WebScraper();</span><br><span class="line"> <span class="keyword">var</span> urlDownloadResult = <span class="keyword">await</span> webscraper.GetContentAsync(docPath);</span><br><span class="line"> <span class="keyword">if</span> (!urlDownloadResult.Success)</span><br><span class="line"> {</span><br><span class="line"> Console.WriteLine(urlDownloadResult.Error);</span><br><span class="line"> <span class="keyword">return</span> <span class="string">""</span>;</span><br><span class="line"> }</span><br><span class="line"> mimeType = urlDownloadResult.ContentType;</span><br><span class="line"> fileBinary = urlDownloadResult.Content;</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">else</span></span><br><span class="line"> {</span><br><span class="line"> mimeType = mimeTypeDetection.GetFileType(docPath);</span><br><span class="line"> <span class="built_in">byte</span>[] fileBytes = File.ReadAllBytes(docPath);</span><br><span class="line"> fileBinary = System.BinaryData.FromBytes(fileBytes);</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">var</span> msExcelDecoderConfig = <span class="keyword">new</span> MsExcelDecoderConfig();</span><br><span class="line"> <span class="keyword">var</span> msPowerPointDecoderConfig = <span class="keyword">new</span> MsPowerPointDecoderConfig();</span><br><span class="line"> <span class="keyword">var</span> decoders = <span class="keyword">new</span> List<IContentDecoder></span><br><span class="line"> {</span><br><span class="line"> <span class="keyword">new</span> TextDecoder(loggerFactory),</span><br><span class="line"> <span class="keyword">new</span> HtmlDecoder(loggerFactory),</span><br><span class="line"> <span class="keyword">new</span> MarkDownDecoder(loggerFactory),</span><br><span class="line"> <span class="keyword">new</span> PdfDecoder(loggerFactory),</span><br><span class="line"> <span class="keyword">new</span> MsWordDecoder(loggerFactory),</span><br><span class="line"> <span class="keyword">new</span> MsExcelDecoder(msExcelDecoderConfig, loggerFactory),</span><br><span class="line"> <span class="keyword">new</span> MsPowerPointDecoder(msPowerPointDecoderConfig, loggerFactory),</span><br><span class="line"> <span class="comment">//new ImageDecoder(ocrEngine, loggerFactory),</span></span><br><span class="line"> };</span><br><span class="line"> <span class="keyword">var</span> decoder = decoders.LastOrDefault(d => d.SupportsMimeType(mimeType));</span><br><span class="line"> <span class="keyword">if</span>(decoder <span class="keyword">is</span> <span class="literal">null</span>)</span><br><span class="line"> {</span><br><span class="line"> Console.WriteLine(<span class="string">$"無法讀取<span class="subst">{mimeType}</span>類型的檔案"</span>);</span><br><span class="line"> <span class="keyword">return</span> <span class="string">""</span>;</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="keyword">var</span> content = <span class="keyword">await</span> decoder.DecodeAsync(fileBinary);</span><br><span class="line"> Console.WriteLine(<span class="string">"File 文字如下 ....."</span>);</span><br><span class="line"> <span class="keyword">var</span> textBuilder = <span class="keyword">new</span> StringBuilder();</span><br><span class="line"> <span class="keyword">foreach</span> (<span class="keyword">var</span> section <span class="keyword">in</span> content.Sections)</span><br><span class="line"> {</span><br><span class="line"> <span class="keyword">var</span> sectionContent = section.Content.Trim();</span><br><span class="line"> <span class="keyword">if</span> (<span class="built_in">string</span>.IsNullOrEmpty(sectionContent)) { <span class="keyword">continue</span>; }</span><br><span class="line"></span><br><span class="line"> textBuilder.Append(sectionContent);</span><br><span class="line"></span><br><span class="line"> <span class="comment">// Add a clean page separation</span></span><br><span class="line"> <span class="keyword">if</span> (section.SentencesAreComplete)</span><br><span class="line"> {</span><br><span class="line"> textBuilder.AppendLineNix();</span><br><span class="line"> textBuilder.AppendLineNix();</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">var</span> fileText = textBuilder.ToString().Trim();</span><br><span class="line"> <span class="keyword">return</span> fileText;</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>3.測試讀取的效果</p><figure class="highlight csharp"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">var</span> docPath = <span class="string">@"new1.docx"</span>;</span><br><span class="line"><span class="keyword">var</span> docBody = <span class="keyword">await</span> ExtractFile(docPath);</span><br><span class="line">Console.WriteLine(<span class="string">$"<span class="subst">{docPath}</span> ========"</span>);</span><br><span class="line">Console.Write(docBody);</span><br><span class="line"></span><br><span class="line"><span class="keyword">var</span> pdfPath = <span class="string">@"pdf1.pdf"</span>;</span><br><span class="line"><span class="keyword">var</span> pdfBody = <span class="keyword">await</span> ExtractFile(pdfPath);</span><br><span class="line">Console.WriteLine(<span class="string">$"<span class="subst">{pdfPath}</span> ========"</span>);</span><br><span class="line">Console.Write(pdfBody);</span><br><span class="line"></span><br><span class="line"><span class="keyword">var</span> urlPath = <span class="string">"https://www.taisugar.com.tw/resting/hualian/CP2.aspx?n=12036"</span>;</span><br><span class="line"><span class="keyword">var</span> urlBody = <span class="keyword">await</span> ExtractFile(urlPath, <span class="literal">true</span>);</span><br><span class="line">Console.WriteLine(<span class="string">$"<span class="subst">{urlBody}</span> ========"</span>);</span><br><span class="line">Console.Write(urlBody);</span><br></pre></td></tr></table></figure><p>結果如下，</p><img src="/2025/03/27/kernel-memory-file-decoder/01.png" class="" title="Docx"><p>讀取 docx 檔案</p><img src="/2025/03/27/kernel-memory-file-decoder/02.png" class="" title="pdf"><p>讀取 pdf 檔案</p><img src="/2025/03/27/kernel-memory-file-decoder/03.png" class="" title="web page"><p>讀取 網頁 內容</p><img src="/2025/03/27/kernel-memory-file-decoder/04.png" class="" title="multi column pdf"><p>讀取 多欄 PDF</p><ul><li>註: 從結果來看，PDF 在 Table 及多欄的資料處理的不是說很好。所以文件儘單欄、簡單 RAG 才會有比較好的效果。</li><li>註: 雖然取出 table 的內容沒有很好，但是在<a href="https://rainmakerho.github.io/2025/03/27/rag-km-mssql/">使用 Kernel Memory 和 MSSQL 快速建立 RAG 服務</a>透過 LLM 來回答倒是正確的內容。</li></ul><h3 id="參考資源"><a href="#參考資源" class="headerlink" title="參考資源"></a>參考資源</h3><p><a href="https://medium.com/@pymupdf/extract-text-from-a-multi-column-document-using-pymupdf-in-python-a0395ebc8e28">Extract Text From a Multi-Column Document Using PyMuPDF in Python</a><br><a href="https://rainmakerho.github.io/2025/03/27/rag-km-mssql/">使用 Kernel Memory 和 MSSQL 快速建立 RAG 服務</a></p>]]></content>
<summary type="html"><h3 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h3><p>在<a href="https://rainmakerho.github.io/2025/02/19/kernel-memory-plain</summary>
<category term="Kernel Memory" scheme="https://rainmakerho.github.io/tags/Kernel-Memory/"/>
<category term="RAG" scheme="https://rainmakerho.github.io/tags/RAG/"/>
<category term="Decoder" scheme="https://rainmakerho.github.io/tags/Decoder/"/>
<category term="TextDecoder" scheme="https://rainmakerho.github.io/tags/TextDecoder/"/>
<category term="HtmlDecoder" scheme="https://rainmakerho.github.io/tags/HtmlDecoder/"/>
<category term="PdfDecoder" scheme="https://rainmakerho.github.io/tags/PdfDecoder/"/>
<category term="MsPowerPointDecoder" scheme="https://rainmakerho.github.io/tags/MsPowerPointDecoder/"/>
<category term="MsWordDecoder" scheme="https://rainmakerho.github.io/tags/MsWordDecoder/"/>
<category term="MarkDownDecoder" scheme="https://rainmakerho.github.io/tags/MarkDownDecoder/"/>
<category term="MsExcelDecoder" scheme="https://rainmakerho.github.io/tags/MsExcelDecoder/"/>
</entry>
<entry>
<title>使用 Kernel Memory 和 MSSQL 快速建立 RAG 服務 - DB 研究</title>
<link href="https://rainmakerho.github.io/2025/03/27/rag-km-mssql-2/"/>
<id>https://rainmakerho.github.io/2025/03/27/rag-km-mssql-2/</id>
<published>2025-03-27T06:30:31.000Z</published>
<updated>2025-04-02T01:47:55.512Z</updated>
<content type="html"><![CDATA[<h3 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h3><p>在前一篇<a href="https://rainmakerho.github.io/2025/03/27/rag-km-mssql/">使用 Kernel Memory 和 MSSQL 快速建立 RAG 服務</a>可以很快建立出 RAG 服務，並讓使用者進行問答。<br>接下來我們來看一下它在 SQLServer 中是如何處理</p><h3 id="研究"><a href="#研究" class="headerlink" title="研究"></a>研究</h3><h5 id="ImportDocumentAsync"><a href="#ImportDocumentAsync" class="headerlink" title="ImportDocumentAsync"></a>ImportDocumentAsync</h5><p>從 <code>await kmClient.ImportDocumentAsync(docPath, index: tenantId, documentId: documentId);</code>開始，<br>它會呼叫 Kernel Memory 的<code>/upload</code> API，<br>呼叫<code>MemoryServerless.ImportDocumentAsync</code>(因為<code>OrchestrationType</code> 設定成 <code>InProcess</code>)，<br>呼叫<code>BaseOrchestrator.ImportDocumentAsync</code>執行<strong>Pipeline</strong>的處理，<br>因為沒設定 Pipeline 自定的 Steps，所以會使用預設的 Steps，<br>預設的 Steps 為<code>extract</code>(讀取文件內容), <code>partition</code>(將文件內容切成多個 Chunks), <code>gen_embeddings</code>(將各 Chunks 轉成 Embedding) 及 <code>save_records</code>(將 Chunks 及 Embeddings 存到 MSSQL)</p><p>在<strong>save_records</strong>時，會做以下幾個步驟</p><p>1.建立基本 Tables<br>建立<code>KMCollections</code>及<code>KMMemories</code>(config 中 SqlServer 區段<code>MemoryCollectionTableName</code>及<code>MemoryTableName</code>的設定值)這 2 個 Table(<code>SqlServerMemory.CreateTablesIfNotExistsAsync</code>)。<br><code>KMCollections</code> Table 只有一個<code>id nvarchar(256)</code>這個欄位，是記錄<code>index</code>的值。<br><code>KMMemories</code> Table 會記錄文件 Chunks 的相關內容。</p><img src="/2025/03/27/rag-km-mssql-2/01.png" class="" title="KMMemories Schema"><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><span class="line">IF OBJECT_ID(N<span class="string">'[dbo].[KMCollections]'</span>, N<span class="string">'U'</span>) <span class="keyword">IS</span> <span class="keyword">NULL</span></span><br><span class="line"> <span class="keyword">CREATE</span> <span class="keyword">TABLE</span> [dbo].[KMCollections]</span><br><span class="line"> ( [id] NVARCHAR(<span class="number">256</span>) <span class="keyword">NOT</span> <span class="keyword">NULL</span>,</span><br><span class="line"> <span class="keyword">PRIMARY</span> KEY ([id])</span><br><span class="line"> );</span><br><span class="line"></span><br><span class="line">IF OBJECT_ID(N<span class="string">'[dbo].[KMMemories]'</span>, N<span class="string">'U'</span>) <span class="keyword">IS</span> <span class="keyword">NULL</span></span><br><span class="line"> <span class="keyword">CREATE</span> <span class="keyword">TABLE</span> [dbo].[KMMemories]</span><br><span class="line"> ( [id] UNIQUEIDENTIFIER <span class="keyword">NOT</span> <span class="keyword">NULL</span>,</span><br><span class="line"> [key] NVARCHAR(<span class="number">256</span>) <span class="keyword">NOT</span> <span class="keyword">NULL</span>,</span><br><span class="line"> [collection] NVARCHAR(<span class="number">256</span>) <span class="keyword">NOT</span> <span class="keyword">NULL</span>,</span><br><span class="line"> [payload] NVARCHAR(MAX),</span><br><span class="line"> [tags] NVARCHAR(MAX),</span><br><span class="line"> [embedding] NVARCHAR(MAX),</span><br><span class="line"> <span class="keyword">PRIMARY</span> KEY ([id]),</span><br><span class="line"> <span class="keyword">FOREIGN</span> KEY ([collection]) <span class="keyword">REFERENCES</span> [dbo].[KMCollections]([id]) <span class="keyword">ON</span> <span class="keyword">DELETE</span> CASCADE,</span><br><span class="line"> <span class="keyword">CONSTRAINT</span> UK_KMMemories <span class="keyword">UNIQUE</span>([collection], [key])</span><br><span class="line"> );</span><br></pre></td></tr></table></figure><p>2.新增<strong>index</strong>資料及建立<strong>index</strong>所需的 Tables(<code>DefaultQueryProvider.PrepareCreateIndexQuery</code>)<br>2.1.將<strong>index</strong>(<code>gss</code>)新增到<code>KMCollections</code>Table 中<br>2.2.建立<code>KMEmbeddings_[index]</code>及<code>KMMemoriesTags_[index]</code>(config 中 SqlServer 區段<code>EmbeddingsTableName</code>及<code>TagsTableName</code>的設定值)這 2 個 Table。(如果不存在才建立)</p><img src="/2025/03/27/rag-km-mssql-2/02.png" class="" title="Kernel Memory Tables"><p>index 相關的 Table 是將<strong>Tags</strong>及<strong>Embeddings</strong>每項存成一筆資料。</p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">BEGIN</span> TRANSACTION;</span><br><span class="line"></span><br><span class="line"> <span class="keyword">INSERT</span> <span class="keyword">INTO</span> [dbo].[KMCollections]([id])</span><br><span class="line"> <span class="keyword">VALUES</span> (<span class="string">'gss'</span>);</span><br><span class="line"></span><br><span class="line"> IF OBJECT_ID(N<span class="string">''</span>[dbo].[KMMemoriesTags_gss]<span class="string">''</span>, N<span class="string">''</span>U<span class="string">''</span>) <span class="keyword">IS</span> <span class="keyword">NULL</span></span><br><span class="line"> <span class="keyword">CREATE</span> <span class="keyword">TABLE</span> [dbo].[KMMemoriesTags_gss]</span><br><span class="line"> (</span><br><span class="line"> [memory_id] UNIQUEIDENTIFIER <span class="keyword">NOT</span> <span class="keyword">NULL</span>,</span><br><span class="line"> [name] NVARCHAR(<span class="number">256</span>) <span class="keyword">NOT</span> <span class="keyword">NULL</span>,</span><br><span class="line"> [<span class="keyword">value</span>] NVARCHAR(<span class="number">256</span>) <span class="keyword">NOT</span> <span class="keyword">NULL</span>,</span><br><span class="line"> <span class="keyword">FOREIGN</span> KEY ([memory_id]) <span class="keyword">REFERENCES</span> [dbo].[KMMemories]([id])</span><br><span class="line"> );</span><br><span class="line"></span><br><span class="line"> IF OBJECT_ID(N<span class="string">''</span>[dbo].[KMEmbeddings_gss]<span class="string">''</span>, N<span class="string">''</span>U<span class="string">''</span>) <span class="keyword">IS</span> <span class="keyword">NULL</span></span><br><span class="line"> <span class="keyword">CREATE</span> <span class="keyword">TABLE</span> [dbo].[KMEmbeddings_gss]</span><br><span class="line"> (</span><br><span class="line"> [memory_id] UNIQUEIDENTIFIER <span class="keyword">NOT</span> <span class="keyword">NULL</span>,</span><br><span class="line"> [vector_value_id] [<span class="type">int</span>] <span class="keyword">NOT</span> <span class="keyword">NULL</span>,</span><br><span class="line"> [vector_value] [<span class="type">float</span>] <span class="keyword">NOT</span> <span class="keyword">NULL</span>,</span><br><span class="line"> <span class="keyword">FOREIGN</span> KEY ([memory_id]) <span class="keyword">REFERENCES</span> [dbo].[KMMemories]([id])</span><br><span class="line"> );</span><br><span class="line"></span><br><span class="line"> IF OBJECT_ID(N<span class="string">''</span>[dbo.IXC_KMEmbeddings_gss]<span class="string">''</span>, N<span class="string">''</span>U<span class="string">''</span>) <span class="keyword">IS</span> <span class="keyword">NULL</span></span><br><span class="line"> <span class="keyword">CREATE</span> CLUSTERED COLUMNSTORE INDEX [IXC_KMEmbeddings_gss]</span><br><span class="line"> <span class="keyword">ON</span> [dbo].[KMEmbeddings_gss];</span><br><span class="line"></span><br><span class="line"><span class="keyword">COMMIT</span>;</span><br></pre></td></tr></table></figure><p>3.刪除<code>documentId</code>為 hr001 的資料(如果有的話)<br>讀取<code>_files/gss/hr001/__pipeline_status.json</code>將舊的 Embeddings 資料刪除(<strong>artifact_type</strong>為<strong>TextEmbeddingVector</strong>的資料)。<br>Loop 去執行刪除，</p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">BEGIN</span> TRANSACTION;</span><br><span class="line"></span><br><span class="line"> <span class="keyword">DELETE</span> [tags]</span><br><span class="line"> <span class="keyword">FROM</span> [dbo].[KMMemoriesTags_gss] [tags]</span><br><span class="line"> <span class="keyword">INNER</span> <span class="keyword">JOIN</span> [dbo].[KMMemories] <span class="keyword">ON</span> [tags].[memory_id] <span class="operator">=</span> [dbo].[KMMemories].[id]</span><br><span class="line"> <span class="keyword">WHERE</span></span><br><span class="line"> [dbo].[KMMemories].[collection] <span class="operator">=</span> N<span class="string">'gss'</span></span><br><span class="line"> <span class="keyword">AND</span> [dbo].[KMMemories].[key]<span class="operator">=</span>N<span class="string">'d=hr001//p=每段的Id'</span>;</span><br><span class="line"></span><br><span class="line"> <span class="keyword">DELETE</span> [embeddings]</span><br><span class="line"> <span class="keyword">FROM</span> [dbo].[KMEmbeddings_gss] [embeddings]</span><br><span class="line"> <span class="keyword">INNER</span> <span class="keyword">JOIN</span> [dbo].[KMMemories] <span class="keyword">ON</span> [embeddings].[memory_id] <span class="operator">=</span> [dbo].[KMMemories].[id]</span><br><span class="line"> <span class="keyword">WHERE</span></span><br><span class="line"> [dbo].[KMMemories].[collection] <span class="operator">=</span> N<span class="string">'gss'</span></span><br><span class="line"> <span class="keyword">AND</span> [dbo].[KMMemories].[key]<span class="operator">=</span>N<span class="string">'d=hr001//p=每段的Id'</span>;</span><br><span class="line"></span><br><span class="line"> <span class="keyword">DELETE</span> <span class="keyword">FROM</span> [dbo].[KMMemories]</span><br><span class="line"> <span class="keyword">WHERE</span> [collection] <span class="operator">=</span> N<span class="string">'gss'</span> <span class="keyword">AND</span> [key]<span class="operator">=</span>N<span class="string">'d=hr001//p=每段的Id'</span>;</span><br><span class="line"></span><br><span class="line"><span class="keyword">COMMIT</span>;</span><br></pre></td></tr></table></figure><p>4.將 Chunks 資料新增/修改到<code>KMMemories</code>,<code>KMEmbeddings_gss</code>及<code>KMMemoriesTags_gss</code> 資料表<br>Loop 執行</p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">BEGIN</span> TRANSACTION;</span><br><span class="line"> <span class="comment">-- 每段 Chunks 資料</span></span><br><span class="line"> <span class="keyword">MERGE</span> <span class="keyword">INTO</span> [dbo].[KMMemories]</span><br><span class="line"> <span class="keyword">USING</span> (<span class="keyword">SELECT</span> N<span class="string">'d=hr001//p=每段的Id'</span>) <span class="keyword">as</span> [src]([key])</span><br><span class="line"> <span class="keyword">ON</span> [dbo].[KMMemories].[key] <span class="operator">=</span> [src].[key]</span><br><span class="line"> <span class="keyword">WHEN</span> MATCHED <span class="keyword">THEN</span></span><br><span class="line"> <span class="keyword">UPDATE</span> <span class="keyword">SET</span> payload<span class="operator">=</span>N<span class="string">'{"url":"","schema":"20231218A","file":"hr001.docx","text":"Chunk 的文字內容，會以u524D","vector_provider":"AI.OpenAI.OpenAITextEmbeddingGenerator","vector_generator":"__","last_update":"2025-03-07T06:20:04"}'</span>, embedding<span class="operator">=</span>N<span class="string">'[Embeddings 值, 如0.01507735,0.013598721]'</span>, tags<span class="operator">=</span>N<span class="string">'{"__document_id":["hr001"],"__file_type":["application/..."],"__file_id":["6ecf..."],"__file_part":["11c5..."],"__part_n":["0"],"__sect_n":["0"]}'</span></span><br><span class="line"> <span class="keyword">WHEN</span> <span class="keyword">NOT</span> MATCHED <span class="keyword">THEN</span></span><br><span class="line"> <span class="keyword">INSERT</span> ([id], [key], [collection], [payload], [tags], [embedding])</span><br><span class="line"> <span class="keyword">VALUES</span> (NEWID(), N<span class="string">'d=hr001//p=每段的Id'</span>, N<span class="string">'gss'</span>, N<span class="string">'{"url":"","schema":"20231218A","file":"hr001.docx","text":"Chunk 的文字內容，會以u524D","vector_provider":"AI.OpenAI.OpenAITextEmbeddingGenerator","vector_generator":"__","last_update":"2025-03-07T06:20:04"}'</span>, N<span class="string">'{"__document_id":["hr001"],"__file_type":["application/..."],"__file_id":["6ecf..."],"__file_part":["11c5..."],"__part_n":["0"],"__sect_n":["0"]}'</span>, N<span class="string">'[Embeddings 值, 如0.01507735,0.013598721]'</span>);</span><br><span class="line"></span><br><span class="line"> <span class="comment">--將每個 Embedding 存到 KMEmbeddings_gss 中</span></span><br><span class="line"> <span class="keyword">MERGE</span> [dbo].[KMEmbeddings_gss] <span class="keyword">AS</span> [tgt]</span><br><span class="line"> <span class="keyword">USING</span> (</span><br><span class="line"> <span class="keyword">SELECT</span></span><br><span class="line"> [dbo].[KMMemories].[id],</span><br><span class="line"> <span class="built_in">cast</span>([vector].[key] <span class="keyword">AS</span> <span class="type">INT</span>) <span class="keyword">AS</span> [vector_value_id],</span><br><span class="line"> <span class="built_in">cast</span>([vector].[<span class="keyword">value</span>] <span class="keyword">AS</span> <span class="type">FLOAT</span>) <span class="keyword">AS</span> [vector_value]</span><br><span class="line"> <span class="keyword">FROM</span> [dbo].[KMMemories]</span><br><span class="line"> <span class="keyword">CROSS</span> APPLY</span><br><span class="line"> openjson(N<span class="string">'[Embeddings 值, 如0.01507735,0.013598721]'</span>) [vector]</span><br><span class="line"> <span class="keyword">WHERE</span> [dbo].[KMMemories].[key] <span class="operator">=</span> N<span class="string">'d=hr001//p=每段的Id'</span></span><br><span class="line"> <span class="keyword">AND</span> [dbo].[KMMemories].[collection] <span class="operator">=</span> N<span class="string">'gss'</span></span><br><span class="line"> ) <span class="keyword">AS</span> [src]</span><br><span class="line"> <span class="keyword">ON</span> [tgt].[memory_id] <span class="operator">=</span> [src].[id] <span class="keyword">AND</span> [tgt].[vector_value_id] <span class="operator">=</span> [src].[vector_value_id]</span><br><span class="line"> <span class="keyword">WHEN</span> MATCHED <span class="keyword">THEN</span></span><br><span class="line"> <span class="keyword">UPDATE</span> <span class="keyword">SET</span> [tgt].[vector_value] <span class="operator">=</span> [src].[vector_value]</span><br><span class="line"> <span class="keyword">WHEN</span> <span class="keyword">NOT</span> MATCHED <span class="keyword">THEN</span></span><br><span class="line"> <span class="keyword">INSERT</span> ([memory_id], [vector_value_id], [vector_value])</span><br><span class="line"> <span class="keyword">VALUES</span> ([src].[id],</span><br><span class="line"> [src].[vector_value_id],</span><br><span class="line"> [src].[vector_value] );</span><br><span class="line"></span><br><span class="line"> <span class="keyword">DELETE</span> <span class="keyword">FROM</span> [tgt]</span><br><span class="line"> <span class="keyword">FROM</span> [dbo].[KMMemoriesTags_gss] <span class="keyword">AS</span> [tgt]</span><br><span class="line"> <span class="keyword">INNER</span> <span class="keyword">JOIN</span> [dbo].[KMMemories] <span class="keyword">ON</span> [tgt].[memory_id] <span class="operator">=</span> [dbo].[KMMemories].[id]</span><br><span class="line"> <span class="keyword">WHERE</span> [dbo].[KMMemories].[key] <span class="operator">=</span> N<span class="string">'d=hr001//p=每段的Id'</span></span><br><span class="line"> <span class="keyword">AND</span> [dbo].[KMMemories].[collection] <span class="operator">=</span> N<span class="string">'gss'</span>;</span><br><span class="line"></span><br><span class="line"> <span class="comment">--把Tags存到 KMMemoriesTags_gss</span></span><br><span class="line"> <span class="keyword">MERGE</span> [dbo].[KMMemoriesTags_gss] <span class="keyword">AS</span> [tgt]</span><br><span class="line"> <span class="keyword">USING</span> (</span><br><span class="line"> <span class="keyword">SELECT</span></span><br><span class="line"> [dbo].[KMMemories].[id],</span><br><span class="line"> <span class="built_in">cast</span>([tags].[key] <span class="keyword">AS</span> NVARCHAR(MAX)) <span class="keyword">COLLATE</span> SQL_Latin1_General_CP1_CI_AS <span class="keyword">AS</span> [tag_name],</span><br><span class="line"> [tag_value].[<span class="keyword">value</span>] <span class="keyword">AS</span> [<span class="keyword">value</span>]</span><br><span class="line"> <span class="keyword">FROM</span> [dbo].[KMMemories]</span><br><span class="line"> <span class="keyword">CROSS</span> APPLY openjson(N<span class="string">'{"__document_id":["hr001"],"__file_type":["application/..."],"__file_id":["6ecf..."],"__file_part":["11c5..."],"__part_n":["0"],"__sect_n":["0"]}'</span>) [tags]</span><br><span class="line"> <span class="keyword">CROSS</span> APPLY openjson(<span class="built_in">cast</span>([tags].[<span class="keyword">value</span>] <span class="keyword">AS</span> NVARCHAR(MAX)) <span class="keyword">COLLATE</span> SQL_Latin1_General_CP1_CI_AS) [tag_value]</span><br><span class="line"> <span class="keyword">WHERE</span> [dbo].[KMMemories].[key] <span class="operator">=</span> N<span class="string">'d=hr001//p=每段的Id'</span></span><br><span class="line"> <span class="keyword">AND</span> [dbo].[KMMemories].[collection] <span class="operator">=</span> N<span class="string">'gss'</span></span><br><span class="line"> ) <span class="keyword">AS</span> [src]</span><br><span class="line"> <span class="keyword">ON</span> [tgt].[memory_id] <span class="operator">=</span> [src].[id] <span class="keyword">AND</span> [tgt].[name] <span class="operator">=</span> [src].[tag_name]</span><br><span class="line"> <span class="keyword">WHEN</span> MATCHED <span class="keyword">THEN</span></span><br><span class="line"> <span class="keyword">UPDATE</span> <span class="keyword">SET</span> [tgt].[<span class="keyword">value</span>] <span class="operator">=</span> [src].[<span class="keyword">value</span>]</span><br><span class="line"> <span class="keyword">WHEN</span> <span class="keyword">NOT</span> MATCHED <span class="keyword">THEN</span></span><br><span class="line"> <span class="keyword">INSERT</span> ([memory_id], [name], [<span class="keyword">value</span>])</span><br><span class="line"> <span class="keyword">VALUES</span> ([src].[id],</span><br><span class="line"> [src].[tag_name],</span><br><span class="line"> [src].[<span class="keyword">value</span>]);</span><br><span class="line"><span class="keyword">COMMIT</span>;</span><br></pre></td></tr></table></figure><h5 id="AskStreamingAsync"><a href="#AskStreamingAsync" class="headerlink" title="AskStreamingAsync"></a>AskStreamingAsync</h5><p>從 <code>kmClient.AskStreamingAsync(question, minRelevance: 0.7, options: new SearchOptions { Stream = true }, index: tenantId)</code>開始，先將問題轉成 Embeddings ，再到資料庫找出相似度大於<code>0.7</code>的 Chunks 資料，</p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">WITH</span></span><br><span class="line">[embedding] <span class="keyword">as</span></span><br><span class="line">(</span><br><span class="line"> <span class="keyword">SELECT</span></span><br><span class="line"> <span class="built_in">cast</span>([key] <span class="keyword">AS</span> <span class="type">INT</span>) <span class="keyword">AS</span> [vector_value_id],</span><br><span class="line"> <span class="built_in">cast</span>([<span class="keyword">value</span>] <span class="keyword">AS</span> <span class="type">FLOAT</span>) <span class="keyword">AS</span> [vector_value]</span><br><span class="line"> <span class="keyword">FROM</span></span><br><span class="line"> openjson(N<span class="string">'[問題的 Embeddings, 例如 0.008542997,0.012275496]'</span>)</span><br><span class="line">),</span><br><span class="line">[similarity] <span class="keyword">AS</span></span><br><span class="line">(</span><br><span class="line"> <span class="keyword">SELECT</span> TOP (<span class="number">100</span>)</span><br><span class="line"> [dbo].[KMEmbeddings_gss].[memory_id],</span><br><span class="line"> <span class="built_in">SUM</span>([embedding].[vector_value] <span class="operator">*</span> [dbo].[KMEmbeddings_gss].[vector_value]) <span class="operator">/</span></span><br><span class="line"> (</span><br><span class="line"> <span class="built_in">SQRT</span>(<span class="built_in">SUM</span>([embedding].[vector_value] <span class="operator">*</span> [embedding].[vector_value]))</span><br><span class="line"> <span class="operator">*</span></span><br><span class="line"> <span class="built_in">SQRT</span>(<span class="built_in">SUM</span>([dbo].[KMEmbeddings_gss].[vector_value] <span class="operator">*</span> [dbo].[KMEmbeddings_gss].[vector_value]))</span><br><span class="line"> ) <span class="keyword">AS</span> cosine_similarity</span><br><span class="line"> <span class="comment">-- sum([embedding].[vector_value] * [dbo].[KMEmbeddings_gss].[vector_value]) as cosine_distance -- Optimized as per https://platform.openai.com/docs/guides/embeddings/which-distance-function-should-i-use</span></span><br><span class="line"> <span class="keyword">FROM</span></span><br><span class="line"> [embedding]</span><br><span class="line"> <span class="keyword">INNER</span> <span class="keyword">JOIN</span></span><br><span class="line"> [dbo].[KMEmbeddings_gss] <span class="keyword">ON</span> [embedding].vector_value_id <span class="operator">=</span> [dbo].[KMEmbeddings_gss].vector_value_id</span><br><span class="line"> <span class="keyword">INNER</span> <span class="keyword">JOIN</span></span><br><span class="line"> [dbo].[KMMemories] <span class="keyword">ON</span> [dbo].[KMEmbeddings_gss].[memory_id] <span class="operator">=</span> [dbo].[KMMemories].[id]</span><br><span class="line"> <span class="keyword">WHERE</span> <span class="number">1</span><span class="operator">=</span><span class="number">1</span></span><br><span class="line"></span><br><span class="line"> <span class="keyword">GROUP</span> <span class="keyword">BY</span></span><br><span class="line"> [dbo].[KMEmbeddings_gss].[memory_id]</span><br><span class="line"> <span class="keyword">ORDER</span> <span class="keyword">BY</span></span><br><span class="line"> cosine_similarity <span class="keyword">DESC</span></span><br><span class="line">)</span><br><span class="line"><span class="keyword">SELECT</span> <span class="keyword">DISTINCT</span></span><br><span class="line"> <span class="built_in">JSON_VALUE</span>(payload, <span class="string">'$.text'</span>) <span class="keyword">AS</span> Txt,</span><br><span class="line"> [dbo].[KMMemories].[id],[dbo].[KMMemories].[key],[dbo].[KMMemories].[payload],[dbo].[KMMemories].[tags],</span><br><span class="line"> [similarity].[cosine_similarity]</span><br><span class="line"><span class="keyword">FROM</span></span><br><span class="line"> [similarity]</span><br><span class="line"><span class="keyword">INNER</span> <span class="keyword">JOIN</span></span><br><span class="line"> [dbo].[KMMemories] <span class="keyword">ON</span> [similarity].[memory_id] <span class="operator">=</span> [dbo].[KMMemories].[id]</span><br><span class="line"><span class="keyword">WHERE</span></span><br><span class="line"> [cosine_similarity] <span class="operator">>=</span> <span class="number">0.7</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">ORDER</span> <span class="keyword">BY</span> [cosine_similarity] <span class="keyword">desc</span></span><br></pre></td></tr></table></figure><img src="/2025/03/27/rag-km-mssql-2/03.png" class="" title="查相似度資料"><p>將這些內容組成 Prompt 來生成答案，如下，</p><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line">Facts:</span><br><span class="line">==== [File:請假規則.docx;Relevance:89.3%]:</span><br><span class="line">Chunk 1 的內容....</span><br><span class="line"></span><br><span class="line">==== [File:請假規則.docx;Relevance:87.1%]:</span><br><span class="line">Chunk 2 的內容....</span><br><span class="line"></span><br><span class="line">==== [File:請假規則.docx;Relevance:87.0%]:</span><br><span class="line">假日，不給給薪薪薪)。</span><br><span class="line">Chunk 3 的內容...</span><br><span class="line">==== [File:請假規則.docx;Relevance:86.6%]:</span><br><span class="line">位為為為0.5小時。</span><br><span class="line">Chunk 4 的內容...</span><br><span class="line"></span><br><span class="line">======</span><br><span class="line">Given only the facts above, provide a comprehensive/detailed answer.</span><br><span class="line">You don't know where the knowledge comes from, just answer.</span><br><span class="line">If you don't have sufficient information, reply with 'INFO NOT FOUND'.</span><br><span class="line">Question: 請假6天，需要簽到那位主管核准? 要事前幾天請呢?</span><br><span class="line">Answer:</span><br></pre></td></tr></table></figure><p>所以最後就從 LLM 就依 Search 到的內容來得到結果。</p><h3 id="結論"><a href="#結論" class="headerlink" title="結論"></a>結論</h3><p>embeddings 就是一堆的數值，比較難的是將文字計算出這些 embeddings，但許多 Embeddings Models 已經可以幫我們做到這些事。<br>所以我們可以將這些數值存到 Table 中，並建立<strong>columnstore index</strong>來加快它的查詢，計算出文字的相似度。<br>Cosine similarity [-1,1]的 SQL 如下，</p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">SELECT</span></span><br><span class="line"> <span class="built_in">SUM</span>(a.value <span class="operator">*</span> b.value) <span class="operator">/</span> (</span><br><span class="line"> <span class="built_in">SQRT</span>(<span class="built_in">SUM</span>(a.value <span class="operator">*</span> a.value)) <span class="operator">*</span> <span class="built_in">SQRT</span>(<span class="built_in">SUM</span>(b.value <span class="operator">*</span> b.value))</span><br><span class="line"> ) <span class="keyword">AS</span> cosine_similarity</span><br><span class="line"><span class="keyword">FROM</span></span><br><span class="line"> vectors_values</span><br></pre></td></tr></table></figure><p>數值越接近 1，代表兩個向量越相似。</p><p>詳細請參考<a href="https://devblogs.microsoft.com/azure-sql/vector-similarity-search-with-azure-sql-database-and-openai/">Vector Similarity Search with Azure SQL database and OpenAI</a></p><ul><li>註:如果使用 MSSQL 建議使用 Azure SQL，或是等待 SQL 2025，使用 Vector 資料型態效能比較好，因為我用 LocalDB 來處理，常常會有 SQL Timeout 的狀況</li></ul><h3 id="參考資源"><a href="#參考資源" class="headerlink" title="參考資源"></a>參考資源</h3><p><a href="https://learn.microsoft.com/zh-tw/sql/relational-databases/vectors/vectors-sql-server?view=azuresqldb-current">SQL 資料庫 引擎中的向量概觀</a><br><a href="https://devblogs.microsoft.com/azure-sql/vector-similarity-search-with-azure-sql-database-and-openai/">Vector Similarity Search with Azure SQL database and OpenAI</a><br><a href="https://devblogs.microsoft.com/azure-sql/sql-vector-store-data-ingestion-and-retrieval-using-azure-logic-apps/">RAG with SQL Vector Store: A Low-Code/No-Code Approach using Azure Logic Apps</a></p>]]></content>
<summary type="html"><h3 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h3><p>在前一篇<a href="https://rainmakerho.github.io/2025/03/27/rag-km-mssql/">使</summary>
<category term="MSSQL" scheme="https://rainmakerho.github.io/tags/MSSQL/"/>
<category term="Kernel Memory" scheme="https://rainmakerho.github.io/tags/Kernel-Memory/"/>
<category term="RAG" scheme="https://rainmakerho.github.io/tags/RAG/"/>
<category term="Retrieval-Augmented Generation" scheme="https://rainmakerho.github.io/tags/Retrieval-Augmented-Generation/"/>
<category term="本地模型" scheme="https://rainmakerho.github.io/tags/%E6%9C%AC%E5%9C%B0%E6%A8%A1%E5%9E%8B/"/>
<category term="SQLServer" scheme="https://rainmakerho.github.io/tags/SQLServer/"/>
<category term="cosine similarity" scheme="https://rainmakerho.github.io/tags/cosine-similarity/"/>
</entry>
<entry>
<title>使用 Kernel Memory 和 MSSQL 快速建立 RAG 服務</title>
<link href="https://rainmakerho.github.io/2025/03/27/rag-km-mssql/"/>
<id>https://rainmakerho.github.io/2025/03/27/rag-km-mssql/</id>
<published>2025-03-27T06:21:38.000Z</published>
<updated>2025-03-27T08:17:36.288Z</updated>
<content type="html"><![CDATA[<h3 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h3><p><a href="https://github.com/microsoft/kernel-memory">Kernel Memory</a>可以讓我們快速建立 RAG(Retrieval-Augmented Generation)服務。<br>支持多種文件格式的文本（如 Image、Office、Pdf、Text 和 WebPages）的提取和分塊，並使用 LLM 嵌入生成器提取嵌入，將嵌入保存到 Vector DB 中。</p><p>文件處理到儲存到 Vector DB 的過程，是透過<a href="https://github.com/microsoft/kernel-memory/blob/main/service/Abstractions/Pipeline/DataPipeline.cs#L237">DataPipeline</a>中的<code>Steps</code>來處理，預設的 Pipleline 是 <code>PipelineStepsExtract, PipelineStepsPartition, PipelineStepsGenEmbeddings, PipelineStepsSaveRecords</code>，就是 1.取得檔案內容，2.切 Chunks ，3.轉成嵌入，4.存到 Vector DB 。<br>Vector DB 也可以存到<code>AzureAISearch</code>, <code>Qdrant</code>, <code>Postgres</code>, <code>Redis</code>, <code>SimpleVectorDb</code>及 <code>SqlServer</code> …<br>當這些文件知識庫存到 Vector DB 後，使用者就可以透過 Filter 來進行 RAG 問答及摘要功能。<br>以下使用相容 OpenAI 的本地小模型（llama.cpp），並將嵌入保存到本地的 MSSQL 中。</p><h3 id="實作"><a href="#實作" class="headerlink" title="實作"></a>實作</h3><h5 id="設定-Kernel-Memory-as-a-Service"><a href="#設定-Kernel-Memory-as-a-Service" class="headerlink" title="設定 Kernel Memory as a Service"></a>設定 Kernel Memory as a Service</h5><p>參考<a href="https://github.com/microsoft/kernel-memory/blob/main/service/Service/README.md">Kernel Memory as a Service</a>，直接在本地運行。<br>在跑之前，要調整<strong>Service 專案</strong>中的<code>appsettings.json</code>檔案。<br>使用相容 OpenAI 的地端小模型(llama.cpp)並將嵌入保存到地端的 MSSQL，<br><code>TextGeneratorType</code> 設定為 <code>OpenAI</code><br>在 <strong>DataIngestion</strong> 區段，<br><code>OrchestrationType</code> 設定成 <code>InProcess</code>,<br><code>EmbeddingGenerationEnabled</code> 設定為 <code>true</code>，<br><code>EmbeddingGeneratorTypes</code> 設定為 <code>[ "OpenAI" ]</code>，<br><code>MemoryDbTypes</code> 設定為 <code>[ "SqlServer" ]</code>，<br><code>TextPartitioning</code> 可以依需求去設定<br><strong>Retrieval</strong> 區段，<br><code>EmbeddingGeneratorType</code> 設定為 <code>OpenAI</code><br><code>MemoryDbType</code> 設定為 <code>SqlServer</code></p><p><strong>Services</strong> 區段中，<br><strong>OpenAI</strong>區段設定地端的 url 及 apikey,</p><figure class="highlight json"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line"><span class="attr">"OpenAI"</span><span class="punctuation">:</span> <span class="punctuation">{</span></span><br><span class="line"> <span class="attr">"TextModel"</span><span class="punctuation">:</span> <span class="string">"phi-4-q4"</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"TextModelMaxTokenTotal"</span><span class="punctuation">:</span> <span class="number">16384</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"TextModelTokenizer"</span><span class="punctuation">:</span> <span class="string">""</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"TextGenerationType"</span><span class="punctuation">:</span> <span class="string">"Chat"</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"EmbeddingModel"</span><span class="punctuation">:</span> <span class="string">"e5-large"</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"EmbeddingModelMaxTokenTotal"</span><span class="punctuation">:</span> <span class="number">500</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"EmbeddingModelTokenizer"</span><span class="punctuation">:</span> <span class="string">""</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"APIKey"</span><span class="punctuation">:</span> <span class="string">"sk-EA-xxx你的 apikey"</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"OrgId"</span><span class="punctuation">:</span> <span class="string">""</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"Endpoint"</span><span class="punctuation">:</span> <span class="string">"https://地端的 URL/v1"</span><span class="punctuation">,</span></span><br><span class="line"> <span class="comment">//...</span></span><br><span class="line"><span class="punctuation">}</span></span><br></pre></td></tr></table></figure><p><strong>SimpleFileStorage</strong> 區段中的 <code>StorageType</code> 改成 <code>Disk</code><br><strong>SqlServer</strong> 區段，設定 <code>ConnectionString</code></p><p>設定好後，可以把程式跑起來，Console 的結果如下，</p><img src="/2025/03/27/rag-km-mssql/01.png" class="" title="dotnet run"><p>透過以上的設定，RAG Service 就完成了哦~<br>接下來透過 MemoryWebClient 來檔案成為知識庫，並進行查詢來看看效果怎麼樣。</p><h5 id="透過-MemoryWebClient-來操作"><a href="#透過-MemoryWebClient-來操作" class="headerlink" title="透過 MemoryWebClient 來操作"></a>透過 MemoryWebClient 來操作</h5><p>Kernel Memory Service 跑起來後，<br>當然，您也可以執行 dotnet run setup 來透過詢問來設定 Config 值。<br>接下來透過 MemoryWebClient 這個類別，來將檔案上傳成為知識庫，並進行查詢來看看效果怎麼樣。</p><p>1.加入<code>Microsoft.KernelMemory</code> Nuget 套件</p><p>2.建立 MemoryWebClient Instance</p><figure class="highlight csharp"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">using</span> Microsoft.KernelMemory;</span><br><span class="line"><span class="keyword">using</span> System.Net.Http;</span><br><span class="line"><span class="keyword">var</span> kmServiceEndpoint = <span class="string">"http://localhost:5000/"</span>;</span><br><span class="line"><span class="keyword">var</span> httpClient = <span class="keyword">new</span> HttpClient();</span><br><span class="line">httpClient.Timeout = TimeSpan.FromMinutes(<span class="number">10</span>);</span><br><span class="line"><span class="keyword">var</span> kmClient = <span class="keyword">new</span> MemoryWebClient(kmServiceEndpoint, client:httpClient);</span><br></pre></td></tr></table></figure><p>如果部署到正式環境，請記得調整 kmServiceEndpoint 的值。</p><p>3.呼叫<code>ImportDocumentAsync</code>來匯入文件</p><figure class="highlight csharp"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">var</span> docPath = <span class="string">@"C:\Projects\請假規則.docx"</span>;</span><br><span class="line"><span class="keyword">var</span> tenantId = <span class="string">"gss"</span>;</span><br><span class="line"><span class="keyword">var</span> documentId = <span class="string">"hr001"</span>;</span><br><span class="line"><span class="keyword">await</span> kmClient.ImportDocumentAsync(docPath, index: tenantId, documentId: documentId);</span><br></pre></td></tr></table></figure><p>註: documentId 可以讓它自動產生，或是指定一個值給它(最好給英數字，不要給中文哦~)</p><p>4.透過<code>IsDocumentReadyAsync</code>來查看是否匯入完成</p><figure class="highlight csharp"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">var</span> isDocReady = <span class="keyword">await</span> kmClient.IsDocumentReadyAsync(documentId, tenantId);</span><br><span class="line"><span class="keyword">if</span>(isDocReady)</span><br><span class="line"> Console.WriteLine(<span class="string">"Document ingestion is complete."</span>);</span><br><span class="line"><span class="keyword">else</span></span><br><span class="line"> Console.WriteLine(<span class="string">"Document is not ready yet."</span>);</span><br></pre></td></tr></table></figure><p>5.進行問答</p><figure class="highlight csharp"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">var</span> question = <span class="string">"請假6天，需要簽到那位主管核准? 要事前幾天請呢?"</span>;</span><br><span class="line">Console.WriteLine(<span class="string">$"Question: <span class="subst">{question}</span>"</span>);</span><br><span class="line">Console.Write(<span class="string">"\nAnswer: "</span>);</span><br><span class="line"><span class="keyword">var</span> answerStream = kmClient.AskStreamingAsync(question, minRelevance: <span class="number">0.7</span>,</span><br><span class="line"> options: <span class="keyword">new</span> SearchOptions { Stream = <span class="literal">true</span> }, index: tenantId);</span><br><span class="line"></span><br><span class="line">List<Citation> sources = [];</span><br><span class="line"><span class="keyword">await</span> <span class="keyword">foreach</span> (<span class="keyword">var</span> answer <span class="keyword">in</span> answerStream)</span><br><span class="line">{</span><br><span class="line"> Console.Write(answer.Result);</span><br><span class="line"> sources.AddRange(answer.RelevantSources);</span><br><span class="line"> <span class="keyword">await</span> Task.Delay(<span class="number">5</span>);</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line">Console.WriteLine(<span class="string">"\n\nSources:\n"</span>);</span><br><span class="line"><span class="keyword">foreach</span> (<span class="keyword">var</span> x <span class="keyword">in</span> sources)</span><br><span class="line">{</span><br><span class="line"> Console.WriteLine(x.SourceUrl != <span class="literal">null</span></span><br><span class="line"> ? <span class="string">$" - <span class="subst">{x.SourceUrl}</span> [<span class="subst">{x.Partitions.First().LastUpdate:D}</span>]"</span></span><br><span class="line"> : <span class="string">$" - <span class="subst">{x.SourceName}</span> - <span class="subst">{x.Link}</span> [<span class="subst">{x.Partitions.First().LastUpdate:D}</span>]"</span>);</span><br><span class="line">}</span><br><span class="line"></span><br></pre></td></tr></table></figure><img src="/2025/03/27/rag-km-mssql/02.png" class="" title="Ask"><p>從上圖可以發現，可以從知識庫中找到對應的資訊並回覆給使用者</p><p>在單欄的 PDF Table 也可以查得到哦。<br>例如<a href="https://www.taisugar.com.tw/upload/UserFiles/News/3469/637479550292673371.pdf">花蓮觀光糖廠參考資料</a>，<br>我查詢 7 月份的房價，是可以正常回覆的，如下，</p><img src="/2025/03/27/rag-km-mssql/03.png" class="" title="Ask Table Result"><p>查到的 Chunk 資料如下，</p><img src="/2025/03/27/rag-km-mssql/04.png" class="" title="Chunk"><ul><li><p>註: <a href="https://huggingface.co/spaces/rrg92/text-embeddings-enus">文字轉成嵌入 Demo 網頁</a></p></li><li><p>註: 在本機可以使用<code>OrchestrationType</code>為<code>InProcess</code>,如果是大檔或是正式機，請使用<code>Distributed</code></p></li></ul><h3 id="參考資源"><a href="#參考資源" class="headerlink" title="參考資源"></a>參考資源</h3><p><a href="https://github.com/microsoft/kernel-memory">Kernel Memory</a><br><a href="https://huggingface.co/spaces/rrg92/text-embeddings-enus">rrg92/text-embeddings-enus</a> - 文字轉成嵌入 Demo 網頁<br><a href="https://github.com/microsoft/kernel-memory/issues/1027">The error "Invalid non-ASCII or control character in header: 0x82B1" occurs during the MemoryWebClient.ImportDocumentAsync operation.</a></p>]]></content>
<summary type="html"><h3 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h3><p><a href="https://github.com/microsoft/kernel-memory">Kernel Memory</a></summary>
<category term="MSSQL" scheme="https://rainmakerho.github.io/tags/MSSQL/"/>
<category term="Kernel Memory" scheme="https://rainmakerho.github.io/tags/Kernel-Memory/"/>
<category term="RAG" scheme="https://rainmakerho.github.io/tags/RAG/"/>
<category term="Retrieval-Augmented Generation" scheme="https://rainmakerho.github.io/tags/Retrieval-Augmented-Generation/"/>
<category term="本地模型" scheme="https://rainmakerho.github.io/tags/%E6%9C%AC%E5%9C%B0%E6%A8%A1%E5%9E%8B/"/>
<category term="SQLServer" scheme="https://rainmakerho.github.io/tags/SQLServer/"/>
</entry>
<entry>
<title>在 ConnectionString 中設定 CommandTimeout !!!</title>
<link href="https://rainmakerho.github.io/2025/03/13/set-commandtimeout-in-connectionstring/"/>
<id>https://rainmakerho.github.io/2025/03/13/set-commandtimeout-in-connectionstring/</id>
<published>2025-03-13T06:19:06.000Z</published>
<updated>2025-03-13T06:22:45.080Z</updated>
<content type="html"><![CDATA[<h3 id="問題"><a href="#問題" class="headerlink" title="問題"></a>問題</h3><p><strong>ConnectionTimeout</strong> 是在 <strong>ConnectionString</strong> 之中設定，<br>而 <strong>CommandTimeout</strong> 是設定 Command 物件的 <strong>CommandTimeout</strong> 屬性。<br>CommandTimeout 可以在 ConnectionString 之中設定嗎?</p><h3 id="解法"><a href="#解法" class="headerlink" title="解法"></a>解法</h3><p>如果 Microsoft SQLServer ，並使用<strong>SqlClient</strong>，是可以的哦~<br>在<a href="https://learn.microsoft.com/en-us/dotnet/api/microsoft.data.sqlclient.sqlconnection.connectionstring?view=sqlclient-dotnet-standard-5.2">SqlConnection.ConnectionString Property</a>中有<code>Command Timeout</code>屬性可以設定。預設<code>Command Timeout</code>是<strong>30 秒</strong>。<br>所以連線字串設定如下，<br><code>Data Source=(local);Integrated Security=true;Initial Catalog=Chinook;Command Timeout=300</code><br>這樣就不用在<strong>Command</strong>物件去設定它的<code>CommandTimeout</code>，只要針對個別的需要不同<code>CommandTimeout</code>特別設定就可以了。</p><h3 id="參考資源"><a href="#參考資源" class="headerlink" title="參考資源"></a>參考資源</h3><p><a href="https://learn.microsoft.com/en-us/dotnet/api/microsoft.data.sqlclient.sqlconnection.connectionstring?view=sqlclient-dotnet-standard-5.2">SqlConnection.ConnectionString Property</a><br><a href="https://erikej.github.io/sqlclient/2020/10/26/sqlclient-commandtimeout-preview.html">Setting the command timeout with the latest .NET SqlClient</a></p>]]></content>
<summary type="html"><h3 id="問題"><a href="#問題" class="headerlink" title="問題"></a>問題</h3><p><strong>ConnectionTimeout</strong> 是在 <strong>ConnectionString</strong</summary>
<category term="SqlConnection" scheme="https://rainmakerho.github.io/tags/SqlConnection/"/>
<category term="ConnectionString" scheme="https://rainmakerho.github.io/tags/ConnectionString/"/>
<category term="CommandTimeout" scheme="https://rainmakerho.github.io/tags/CommandTimeout/"/>
<category term="SqlClient" scheme="https://rainmakerho.github.io/tags/SqlClient/"/>
</entry>
<entry>
<title>Assistant API 的 Function Call Arguments 亂碼問題</title>
<link href="https://rainmakerho.github.io/2025/03/13/assistant-api-in-gpt-mis-encodes-function-call-arguments/"/>
<id>https://rainmakerho.github.io/2025/03/13/assistant-api-in-gpt-mis-encodes-function-call-arguments/</id>
<published>2025-03-13T05:51:10.000Z</published>
<updated>2025-03-13T06:15:25.574Z</updated>
<content type="html"><![CDATA[<h3 id="問題"><a href="#問題" class="headerlink" title="問題"></a>問題</h3><p>最近使用 Azure AI 中的 <strong>Assistant API</strong> 時，當它使用到了<strong>Function Call</strong>時，它傳入的參數值居然變成了亂碼!!!<br>似乎本來是 UTF-8 結果被轉成了 Latin-1 編碼。</p><h3 id="解法"><a href="#解法" class="headerlink" title="解法"></a>解法</h3><p>後來找到<a href="https://learn.microsoft.com/en-us/answers/questions/2183705/assistant-api-stream-in-gpt-4o-(2024-11-20)-mis-en">Assistant API Stream in gpt-4o (2024-11-20) mis-encodes function call arguments</a>，<br>原來是因為使用到了最新版本的<strong>GPT-4o</strong>的版本(<code>2024-11-20</code>)。<br>將部署的模型將版本改到前一版本(<code>2024-08-06</code>)，就不會有亂碼的狀況了。</p><h3 id="參考資源"><a href="#參考資源" class="headerlink" title="參考資源"></a>參考資源</h3><p><a href="https://learn.microsoft.com/en-us/answers/questions/2183705/assistant-api-stream-in-gpt-4o-(2024-11-20)-mis-en">Assistant API Stream in gpt-4o (2024-11-20) mis-encodes function call arguments</a></p>]]></content>
<summary type="html"><h3 id="問題"><a href="#問題" class="headerlink" title="問題"></a>問題</h3><p>最近使用 Azure AI 中的 <strong>Assistant API</strong> 時，當它使用到了<strong>Functi</summary>
<category term="亂碼" scheme="https://rainmakerho.github.io/tags/%E4%BA%82%E7%A2%BC/"/>
<category term="AOAI" scheme="https://rainmakerho.github.io/tags/AOAI/"/>
<category term="Assistant API" scheme="https://rainmakerho.github.io/tags/Assistant-API/"/>
<category term="arguments" scheme="https://rainmakerho.github.io/tags/arguments/"/>
<category term="mis-encodes" scheme="https://rainmakerho.github.io/tags/mis-encodes/"/>
<category term="Latin-1" scheme="https://rainmakerho.github.io/tags/Latin-1/"/>
</entry>
<entry>
<title>如何設定讓一般使用者在透過Azure AD登入App時，有權限可以去同意要求的權限</title>
<link href="https://rainmakerho.github.io/2025/03/12/azure-ad-app-consent-permissions/"/>
<id>https://rainmakerho.github.io/2025/03/12/azure-ad-app-consent-permissions/</id>
<published>2025-03-12T04:52:47.000Z</published>
<updated>2025-03-13T05:44:56.383Z</updated>
<content type="html"><![CDATA[<h3 id="問題"><a href="#問題" class="headerlink" title="問題"></a>問題</h3><p>在 Microsoft Entra ID 的 應用程式註冊 (App Registrations)設定好後，<br>一般使用者要透過 Azure AD 登入時，會出現<strong>需要管理員核准</strong>，如下，</p><img src="/2025/03/12/azure-ad-app-consent-permissions/04.jpg" class="" title="Consent and permissions"><p>要怎麼設定才可以讓一般使用者可以自行去同意呢?</p><h3 id="解法"><a href="#解法" class="headerlink" title="解法"></a>解法</h3><h5 id="設定-Consent-and-permissions"><a href="#設定-Consent-and-permissions" class="headerlink" title="設定 Consent and permissions"></a>設定 Consent and permissions</h5><p>到 Microsoft Entra ID=>Enterprise applications=>Consent and permissions(在 Security 區段)<br>原本設定的值是<strong>Do not allow user consent</strong>，所以才會<strong>需要管理員核准</strong>，<br>所以可以將它改成<strong>Allow user consent for apps from verified publishers, for selected permissions (Recommended)<strong>，<br>或是</strong>Allow user consent for apps</strong>，如下圖，</p><img src="/2025/03/12/azure-ad-app-consent-permissions/01.png" class="" title="Consent and permissions"><img src="/2025/03/12/azure-ad-app-consent-permissions/02.png" class="" title="Allow user consent"><h3 id="注意事項"><a href="#注意事項" class="headerlink" title="注意事項"></a>注意事項</h3><p>1.在 應用程式註冊 (App Registrations) 中 App 的 API permissions 如果需要管理員 Grant 的，要先請管理員去 Grant. <strong>Redirect URIs</strong> 也要設定</p><p>2.有些 App 如果在<strong>Enterprise applications</strong>找不到，請在應用程式註冊 (App Registrations) 中 App 的 Overview ，<strong>Managed application in local directory</strong>點選<code>Create Service Principal</code>去建立</p><p>3.如果 App 有在<strong>Enterprise applications</strong>中，但它的<strong>Application type</strong>不是<strong>Enterprise Applications</strong>，請在<strong>Enterprise applications</strong>中將它刪除，再執行上述第 2 點重新建立它就可以了</p><img src="/2025/03/12/azure-ad-app-consent-permissions/03.png" class="" title="Enterprise applications"><ul><li>註: 在 應用程式註冊 (App Registrations) 中 App 的 API permissions 如果需要管理員 Grant ，而沒有 Grant 在登入時，也會出現<strong>需要管理員核准</strong>的訊息。</li></ul><h3 id="參考資源"><a href="#參考資源" class="headerlink" title="參考資源"></a>參考資源</h3><p><a href="https://learn.microsoft.com/en-us/answers/questions/270680/app-registration-vs-enterprise-applications">App Registration vs Enterprise Applications</a><br><a href="https://learn.microsoft.com/en-us/entra/identity-platform/app-objects-and-service-principals">Application and service principal objects in Microsoft Entra ID</a></p>]]></content>
<summary type="html"><h3 id="問題"><a href="#問題" class="headerlink" title="問題"></a>問題</h3><p>在 Microsoft Entra ID 的 應用程式註冊 (App Registrations)設定好後，<br>一般使用者要透過 Azu</summary>
<category term="AzureAD" scheme="https://rainmakerho.github.io/tags/AzureAD/"/>
<category term="Enterprise applications" scheme="https://rainmakerho.github.io/tags/Enterprise-applications/"/>
<category term="Entra ID" scheme="https://rainmakerho.github.io/tags/Entra-ID/"/>
</entry>
<entry>
<title>使用 Kernel Memory 的 PlainTextChunker 來幫我們切 Chunk</title>
<link href="https://rainmakerho.github.io/2025/02/19/kernel-memory-plaintext-chunker/"/>
<id>https://rainmakerho.github.io/2025/02/19/kernel-memory-plaintext-chunker/</id>
<published>2025-02-19T07:19:49.000Z</published>
<updated>2025-02-19T07:35:30.423Z</updated>
<content type="html"><![CDATA[<h3 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h3><p>Kernel Memory 0.97.250211.1 已改用<code>PlainTextChunker</code>來切 Chunk，<br>之前使用<code>TextChunker</code>要設定每行的最大 Token 數及每段的最大 Token。<br>而<code>PlainTextChunker</code>只需要設定每段的最大 Token 數就可以了，以下我們來看看要怎麼透過它來切 Chunks.</p><h3 id="實作"><a href="#實作" class="headerlink" title="實作"></a>實作</h3><p>1.加入<code>Microsoft.KernelMemory</code> Nuget 套件</p><p>2.呼叫<code>Split</code>Method，將文字傳入，並設定每段最大的 Token 數及 Overlap 的數量</p><figure class="highlight csharp"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">using</span> Microsoft.KernelMemory.AI;</span><br><span class="line"><span class="keyword">using</span> Microsoft.KernelMemory.Chunkers;</span><br><span class="line"></span><br><span class="line"><span class="meta">#<span class="keyword">pragma</span> <span class="keyword">warning</span> disable KMEXP00</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">var</span> contents = <span class="string">""</span><span class="string">"</span></span><br><span class="line"><span class="string">... 要切 Chunk 的 文字內容 ...</span></span><br><span class="line"><span class="string">"</span><span class="string">""</span>;</span><br><span class="line"></span><br><span class="line"><span class="keyword">var</span> tokenizer = <span class="keyword">new</span> CL100KTokenizer();</span><br><span class="line"><span class="keyword">var</span> chunker = <span class="keyword">new</span> PlainTextChunker(tokenizer);</span><br><span class="line"><span class="keyword">var</span> options = <span class="keyword">new</span> PlainTextChunkerOptions</span><br><span class="line"> {</span><br><span class="line"> MaxTokensPerChunk = <span class="number">50</span>,</span><br><span class="line"> Overlap = <span class="number">0</span></span><br><span class="line"> };</span><br><span class="line"><span class="keyword">var</span> sentences = chunker.Split(contents, options );</span><br><span class="line"><span class="built_in">int</span> i = <span class="number">0</span>;</span><br><span class="line"><span class="keyword">foreach</span> (<span class="keyword">var</span> sentence <span class="keyword">in</span> sentences)</span><br><span class="line">{</span><br><span class="line"> Console.WriteLine(<span class="string">$" *** 第<span class="subst">{++i}</span>段 ======================"</span>);</span><br><span class="line"> Console.WriteLine(sentence);</span><br><span class="line"> <span class="comment">//呼叫 Embedding Model API, 將段落轉成 Embedding ...</span></span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>改用<code>PlainTextChunker</code>使用上簡單了許多。</p><h3 id="參考資源"><a href="#參考資源" class="headerlink" title="參考資源"></a>參考資源</h3><p><a href="https://github.com/microsoft/kernel-memory">Kernel Memory</a></p>]]></content>
<summary type="html"><h3 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h3><p>Kernel Memory 0.97.250211.1 已改用<code>PlainTextChunker</code>來切 Chunk，<</summary>
<category term="Kernel Memory" scheme="https://rainmakerho.github.io/tags/Kernel-Memory/"/>
<category term="RAG" scheme="https://rainmakerho.github.io/tags/RAG/"/>
<category term="Chunk" scheme="https://rainmakerho.github.io/tags/Chunk/"/>
<category term="KM" scheme="https://rainmakerho.github.io/tags/KM/"/>
<category term="Split" scheme="https://rainmakerho.github.io/tags/Split/"/>
</entry>
<entry>
<title>使用 Kernel Memory 的 TextChunker 來幫我們切 Chunk</title>
<link href="https://rainmakerho.github.io/2025/02/06/kernel-memory-textchunker/"/>
<id>https://rainmakerho.github.io/2025/02/06/kernel-memory-textchunker/</id>
<published>2025-02-06T02:40:50.000Z</published>
<updated>2025-02-19T07:24:08.785Z</updated>
<content type="html"><![CDATA[<h3 id="註-Kernel-Memory-0-97-250211-1-已改用-PlainTextChunker"><a href="#註-Kernel-Memory-0-97-250211-1-已改用-PlainTextChunker" class="headerlink" title="註: Kernel Memory 0.97.250211.1 已改用 PlainTextChunker"></a>註: Kernel Memory 0.97.250211.1 已改用 PlainTextChunker</h3><h3 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h3><p>使用 RAG(Retrieval-Augmented Generation)需要將內容切成 Chunk,<br>而 <a href="https://github.com/microsoft/kernel-memory">Kernel Memory</a> 的 TextChunker 正可以幫我們來做這件事。</p><h3 id="實作"><a href="#實作" class="headerlink" title="實作"></a>實作</h3><p>1.加入<code>Microsoft.KernelMemory</code> Nuget 套件</p><p>2.讀取文件內容後，交給<code>TextChunker</code> 來處理</p><figure class="highlight csharp"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">using</span> Microsoft.KernelMemory.AI;</span><br><span class="line"><span class="keyword">using</span> Microsoft.KernelMemory.DataFormats.Text;</span><br><span class="line"></span><br><span class="line"><span class="meta">#<span class="keyword">pragma</span> <span class="keyword">warning</span> disable KMEXP00</span></span><br><span class="line"><span class="built_in">string</span> contents = System.IO.File.ReadAllText(<span class="string">@"你的文字檔"</span>);</span><br><span class="line">TextChunker.TokenCounter tokenCounter = <span class="keyword">new</span> CL100KTokenizer().CountTokens;</span><br><span class="line"></span><br><span class="line"><span class="comment">//計算字串的Token數</span></span><br><span class="line"><span class="built_in">int</span> tokenCount = tokenCounter(contents);</span><br><span class="line">Console.WriteLine(<span class="string">$"The text contains <span class="subst">{tokenCount}</span> tokens."</span>);</span><br><span class="line"></span><br><span class="line"><span class="comment">//每行最多100個Token，超過就換到下一行</span></span><br><span class="line"><span class="built_in">int</span> maxTokensPerLine = <span class="number">100</span>;</span><br><span class="line"><span class="keyword">var</span> sentences = TextChunker.SplitPlainTextLines(contents, maxTokensPerLine: maxTokensPerLine, tokenCounter: tokenCounter);</span><br><span class="line">Console.WriteLine(<span class="string">"======= SplitPlainTextLines ======="</span>);</span><br><span class="line"><span class="built_in">int</span> i = <span class="number">0</span>;</span><br><span class="line"><span class="keyword">foreach</span> (<span class="keyword">var</span> sentence <span class="keyword">in</span> sentences) {</span><br><span class="line"> Console.WriteLine(<span class="string">$"<span class="subst">{++i}</span>=><span class="subst">{sentence}</span>"</span>);</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="comment">//每個段落最多400個Token，超過就換到下一個段落</span></span><br><span class="line"><span class="built_in">int</span> maxTokensPerParagraph = <span class="number">400</span>;</span><br><span class="line"><span class="built_in">int</span> overlapTokens = <span class="number">10</span>;</span><br><span class="line"><span class="keyword">var</span> partitions = TextChunker.SplitPlainTextParagraphs(sentences,</span><br><span class="line"> maxTokensPerParagraph: maxTokensPerParagraph, overlapTokens: overlapTokens, tokenCounter: tokenCounter);</span><br><span class="line">Console.WriteLine(<span class="string">"======= SplitPlainTextParagraphs ======="</span>);</span><br><span class="line">i = <span class="number">0</span>;</span><br><span class="line"><span class="keyword">foreach</span> (<span class="keyword">var</span> partition <span class="keyword">in</span> partitions)</span><br><span class="line">{</span><br><span class="line"> Console.WriteLine(<span class="string">$"<span class="subst">{++i}</span>=><span class="subst">{partition}</span>"</span>);</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">//最後將這些 partition 呼叫 Embedding Model ，轉成 Vector 一併儲起來</span></span><br><span class="line"></span><br></pre></td></tr></table></figure><ul><li>註: maxTokensPerLine, maxTokensPerParagraph, overlapTokens 請依文件內容來進行調整。</li></ul><h3 id="參考資源"><a href="#參考資源" class="headerlink" title="參考資源"></a>參考資源</h3><p><a href="https://github.com/microsoft/kernel-memory">Kernel Memory</a></p>]]></content>
<summary type="html"><h3 id="註-Kernel-Memory-0-97-250211-1-已改用-PlainTextChunker"><a href="#註-Kernel-Memory-0-97-250211-1-已改用-PlainTextChunker" class="headerlink"</summary>
<category term="Kernel Memory" scheme="https://rainmakerho.github.io/tags/Kernel-Memory/"/>
<category term="RAG" scheme="https://rainmakerho.github.io/tags/RAG/"/>
<category term="TextChunker" scheme="https://rainmakerho.github.io/tags/TextChunker/"/>
<category term="SplitPlainTextLines" scheme="https://rainmakerho.github.io/tags/SplitPlainTextLines/"/>
<category term="SplitPlainTextParagraphs" scheme="https://rainmakerho.github.io/tags/SplitPlainTextParagraphs/"/>
</entry>
<entry>
<title>ABP 如何客製 Blazor Server 的 Login Page</title>
<link href="https://rainmakerho.github.io/2025/01/24/abp-blazor-custom-login/"/>
<id>https://rainmakerho.github.io/2025/01/24/abp-blazor-custom-login/</id>
<published>2025-01-24T07:13:27.000Z</published>
<updated>2025-01-24T07:47:43.000Z</updated>
<content type="html"><![CDATA[<h3 id="問題"><a href="#問題" class="headerlink" title="問題"></a>問題</h3><p>想要在 ABP Blazor Server 專案中客製 Login 頁面要怎麼做呢?</p><h3 id="實作"><a href="#實作" class="headerlink" title="實作"></a>實作</h3><p>ABP Blazor 登入是使用 Razor Page，</p><p>1.在 Blazor 專案的<strong>Pages</strong>目錄中，新增<strong>Account</strong>目錄，並新增<strong>Login.cshtml</strong>，如下，</p><img src="/2025/01/24/abp-blazor-custom-login/01.png" class="" title="add Login.cshtml"><p>請注意，是專案的<strong>Pages</strong>目錄，不是<strong>Components/Pages</strong>目錄哦~~</p><p>2.在<strong>Login.cshtml.cs</strong>中，修改 Model 名稱，並繼承自<code>Volo.Abp.Account.Web.Pages.Account.LoginModel</code><br>並將預設的<code>OnGet</code> Method 刪除掉，如下，</p><figure class="highlight csharp"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">public</span> <span class="keyword">class</span> <span class="title">CustomLoginModel</span> : <span class="title">LoginModel</span></span><br><span class="line">{</span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="title">CustomLoginModel</span>(<span class="params"></span></span></span><br><span class="line"><span class="params"><span class="function"> IAuthenticationSchemeProvider schemeProvider,</span></span></span><br><span class="line"><span class="params"><span class="function"> IOptions<AbpAccountOptions> accountOptions,</span></span></span><br><span class="line"><span class="params"><span class="function"> IOptions<IdentityOptions> identityOptions,</span></span></span><br><span class="line"><span class="params"><span class="function"> IdentityDynamicClaimsPrincipalContributorCache identityDynamicClaimsPrincipalContributorCache</span>)</span></span><br><span class="line"><span class="function"> : <span class="title">base</span>(<span class="params">schemeProvider, accountOptions, identityOptions, identityDynamicClaimsPrincipalContributorCache</span>)</span></span><br><span class="line"> {</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>3.Copy <a href="https://github.com/abpframework/abp/blob/dev/modules/account/src/Volo.Abp.Account.Web/Pages/Account/Login.cshtml">Abp Login.cshtml</a>的內容到<strong>Login.cshtml</strong>中，並將<code>@model</code>改成<code>CustomLoginModel</code>，如下，</p><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><span class="line">@page</span><br><span class="line">@model 專案Blazor.Pages.Account.CustomLoginModel</span><br><span class="line"></span><br><span class="line">@using Microsoft.AspNetCore.Mvc.Localization</span><br><span class="line">@using Volo.Abp.Account.Localization</span><br><span class="line">@using Volo.Abp.Account.Settings</span><br><span class="line">@using Volo.Abp.Account.Web.Pages.Account</span><br><span class="line">@using Volo.Abp.AspNetCore.Mvc.UI.Theming</span><br><span class="line">@using Volo.Abp.Identity;</span><br><span class="line">@using Volo.Abp.Settings</span><br><span class="line"></span><br><span class="line">@inject IHtmlLocalizer<AccountResource> L</span><br><span class="line">@inject IThemeManager ThemeManager</span><br><span class="line">@inject Volo.Abp.Settings.ISettingProvider SettingProvider</span><br><span class="line"></span><br><span class="line">... 其他的 內容</span><br></pre></td></tr></table></figure><p>4.在<strong>Pages</strong>目錄中，新增<code>_ViewImports.cshtml</code>，並刪除<code>ViewImports.cshtml.cs</code>檔案<br>在<code>_ViewImports.cshtml</code>中加入 TagHelper 設定，</p><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">@addTagHelper *, Microsoft.AspNetCore.Mvc.TagHelpers</span><br><span class="line">@addTagHelper *, Volo.Abp.AspNetCore.Mvc.UI</span><br><span class="line">@addTagHelper *, Volo.Abp.AspNetCore.Mvc.UI.Bootstrap</span><br><span class="line">@addTagHelper *, Volo.Abp.AspNetCore.Mvc.UI.Bundling</span><br></pre></td></tr></table></figure><p>所以最後會新增<code>Login.cshtml</code> 及 <code>_ViewImports.cshtml</code> ，如下，</p><img src="/2025/01/24/abp-blazor-custom-login/02.png" class="" title="Projects"><p>再運行系統來檢查一下，切到登入頁面來驗證是否有跑到我們的客製登入頁，</p><img src="/2025/01/24/abp-blazor-custom-login/03.png" class="" title="Login"><p>同時檢查一下，驗證看看 Razor Page 的 TagHelper 是否有正常運作，<br>Form 是否有包含<code>__RequestVerificationToken</code>的 Hidden Field，</p><img src="/2025/01/24/abp-blazor-custom-login/04.png" class="" title="Verify Login Html"><h3 id="參考資源"><a href="#參考資源" class="headerlink" title="參考資源"></a>參考資源</h3><p><a href="https://github.com/abpframework/abp/blob/dev/modules/account/src/Volo.Abp.Account.Web/Pages/Account/Login.cshtml">Abp Login.cshtml</a><br><a href="https://abp.io/docs/latest/framework/ui/mvc-razor-pages/customization-user-interface">Abp Overriding a Razor Page</a></p>]]></content>
<summary type="html"><h3 id="問題"><a href="#問題" class="headerlink" title="問題"></a>問題</h3><p>想要在 ABP Blazor Server 專案中客製 Login 頁面要怎麼做呢?</p>
<h3 id="實作"><a href="#實</summary>
<category term="Blazor" scheme="https://rainmakerho.github.io/tags/Blazor/"/>
<category term="Custom" scheme="https://rainmakerho.github.io/tags/Custom/"/>
<category term="Abp" scheme="https://rainmakerho.github.io/tags/Abp/"/>
<category term="Login" scheme="https://rainmakerho.github.io/tags/Login/"/>
<category term="cshtml" scheme="https://rainmakerho.github.io/tags/cshtml/"/>
</entry>
<entry>
<title>ABP 如何使用 Azure Active Directory 驗證登入</title>
<link href="https://rainmakerho.github.io/2025/01/21/abp-use-azure-active-directory-auth/"/>
<id>https://rainmakerho.github.io/2025/01/21/abp-use-azure-active-directory-auth/</id>
<published>2025-01-21T05:09:17.000Z</published>
<updated>2025-01-21T06:13:13.528Z</updated>
<content type="html"><![CDATA[<h3 id="問題"><a href="#問題" class="headerlink" title="問題"></a>問題</h3><p>ABP 預設會使用自己的驗證，如果要使用 Azure AD 驗證登入的話，<br>可以依 <a href="https://abp.io/community/articles/how-to-use-the-azure-active-directory-authentication-for-mvc-razor-page-applications-4603b9cf">How to Use the Azure Active Directory Authentication for MVC / Razor Page Applications</a> ，<br>來啟用 Azure AD OpenIdConnect 驗證登入，以下就一步步地來實作它</p><h3 id="實作"><a href="#實作" class="headerlink" title="實作"></a>實作</h3><p>1.註冊 App<br>到<a href="https://portal.azure.com/">Portal Azure</a>的<strong>Microsoft Entra ID</strong>中，<br>選點 Menu Manage -> App registrations 功能，點選「+ New registration」來註冊 App，</p><img src="/2025/01/21/abp-use-azure-active-directory-auth/01.png" class="" title="New registration"><p>輸入 App 的名稱，選擇 account type(如果是公司內用請選擇<strong>Single tenant</strong>)，<br><strong>Redirect URI (optional)<strong>請選擇</strong>Web</strong>及輸入 Abp 網站的 URL，並加上<strong>signin-oidc</strong>(例如<code>https://localhost:44392/signin-oidc</code>)，然後按下<strong>Register</strong> Button。</p><p>2.產生 Client Secrets<br>在已建立好的 App 中，選點 Menu Manage-> Certificates & secrets<br>點選<strong>Client secrets</strong>，按下「+ New client secret」，輸入 Description，選擇過期時間，再按下<strong>Add</strong></p><img src="/2025/01/21/abp-use-azure-active-directory-auth/02.png" class="" title="New client secret"><p>3.Copy client secret 的值<br>將 client secret 的值 Copy 存下來，如下圖</p><img src="/2025/01/21/abp-use-azure-active-directory-auth/03.png" class="" title="Copy client secret"><p>4.Copy Directory (tenant) ID 及 Application (client) ID<br>點選 Menu Overview ，將 Directory (tenant) ID 及 Application (client) ID Copy 存下來<br>選選<strong>Endpoints</strong>可以知道要驗證的 URL，<br>如果是 Single Tenant ，OAuth 的 endpoint 會是 <code>https://login.microsoftonline.com/{你的TenantId}/oauth2/v2.0</code>，如果是多租戶，原本的 TenantId 則改為 <strong>common</strong></p><img src="/2025/01/21/abp-use-azure-active-directory-auth/04.png" class="" title="App Endpoints"><p>5.在<code>appsettings.json</code>中加入 Azure AD 的 App 設定值，<br>在<a href="https://abp.io/community/articles/how-to-use-the-azure-active-directory-authentication-for-mvc-razor-page-applications-4603b9cf">How to Use the Azure Active Directory Authentication for MVC / Razor Page Applications</a>的<strong>ResponseType</strong>是使用<code>CodeIdToken</code>，這裡我們改用<code>code</code></p><figure class="highlight json"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="attr">"AzureAd"</span><span class="punctuation">:</span> <span class="punctuation">{</span></span><br><span class="line"> <span class="attr">"Instance"</span><span class="punctuation">:</span> <span class="string">"https://login.microsoftonline.com/"</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"TenantId"</span><span class="punctuation">:</span> <span class="string">"common"</span><span class="punctuation">,</span> <span class="comment">//如果是 Single Tenant 請改成 TenantId</span></span><br><span class="line"> <span class="attr">"ClientId"</span><span class="punctuation">:</span> <span class="string">"ClientId Value"</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"ClientSecret"</span><span class="punctuation">:</span> <span class="string">"ClientSecret Value"</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"CallbackPath"</span><span class="punctuation">:</span> <span class="string">"/signin-oidc"</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"ResponseType"</span><span class="punctuation">:</span> <span class="string">"code"</span></span><br><span class="line"><span class="punctuation">}</span></span><br></pre></td></tr></table></figure><p>6.在 Web 專案的<strong>Module</strong>檔案中的<code>ConfigureAuthentication</code> Method 中設定啟用 Azure AD 驗證<br>在 Web 專案加入<code>Microsoft.AspNetCore.Authentication.OpenIdConnect</code>套件，並修改<code>ConfigureAuthentication</code> Method<br><strong>ConfigureAuthentication</strong> Method，因為需要取得<code>appsettings.json</code>中的設定值，所以要多一個<code>IConfiguration configuration</code>的參數</p><figure class="highlight csharp"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">private</span> <span class="keyword">void</span> <span class="title">ConfigureAuthentication</span>(<span class="params">ServiceConfigurationContext context, IConfiguration configuration</span>)</span></span><br><span class="line">{</span><br><span class="line">context.Services.AddAuthentication()</span><br><span class="line">.AddOpenIdConnect(<span class="string">"AzureOpenId"</span>, <span class="string">"Azure Active Directory OpenId"</span>, options =></span><br><span class="line">{</span><br><span class="line">options.Authority = <span class="string">$"<span class="subst">{configuration[<span class="string">"AzureAd:Instance"</span>]}</span><span class="subst">{configuration[<span class="string">"AzureAd:TenantId"</span>]}</span>/v2.0"</span>;</span><br><span class="line"></span><br><span class="line">options.ClientId = configuration[<span class="string">"AzureAd:ClientId"</span>];</span><br><span class="line">options.CallbackPath = configuration[<span class="string">"AzureAd:CallbackPath"</span>];</span><br><span class="line">options.ClientSecret = configuration[<span class="string">"AzureAd:ClientSecret"</span>];</span><br><span class="line">options.ResponseType = configuration[<span class="string">"AzureAd:ResponseType"</span>]!;</span><br><span class="line">options.GetClaimsFromUserInfoEndpoint = <span class="literal">true</span>;</span><br><span class="line">options.SaveTokens = <span class="literal">true</span>;</span><br><span class="line">options.SignInScheme = IdentityConstants.ExternalScheme;</span><br><span class="line">options.TokenValidationParameters = <span class="keyword">new</span> TokenValidationParameters</span><br><span class="line">{</span><br><span class="line">ValidateIssuer = <span class="literal">true</span>,</span><br><span class="line">IssuerValidator = (issuer, securityToken, validationParameters) =></span><br><span class="line">{</span><br><span class="line"><span class="keyword">if</span> (issuer.StartsWith(<span class="string">$"<span class="subst">{configuration[<span class="string">"AzureAd:Instance"</span>]}</span>"</span>) && issuer.EndsWith(<span class="string">"/v2.0"</span>))</span><br><span class="line">{</span><br><span class="line"><span class="keyword">return</span> issuer;</span><br><span class="line">}</span><br><span class="line"><span class="keyword">throw</span> <span class="keyword">new</span> SecurityTokenInvalidIssuerException(<span class="string">$"Invalid issuer: <span class="subst">{issuer}</span>"</span>);</span><br><span class="line">}</span><br><span class="line">};</span><br><span class="line">});</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>7.執行程式進行測試<br>在<strong>登入</strong>頁面中，點下<strong>Azure Active Directory OpenId</strong>Button，<br>就會導到 Azure AD 進行登入驗證，如下圖</p><img src="/2025/01/21/abp-use-azure-active-directory-auth/05.png" class="" title="Login"><img src="/2025/01/21/abp-use-azure-active-directory-auth/06.png" class="" title="Azure AD 登入"><p>Azure AD 登入驗證後，會回到 Abp 程式，進行註冊來將 User 新增到資料庫之中，如下圖</p><img src="/2025/01/21/abp-use-azure-active-directory-auth/07.png" class="" title="註冊"><p>回到首頁後，就可以看到使用者已登入到系統，如下圖</p><img src="/2025/01/21/abp-use-azure-active-directory-auth/08.png" class="" title="首頁"><h3 id="參考資源"><a href="#參考資源" class="headerlink" title="參考資源"></a>參考資源</h3><p><a href="https://abp.io/community/articles/how-to-use-the-azure-active-directory-authentication-for-mvc-razor-page-applications-4603b9cf">How to Use the Azure Active Directory Authentication for MVC / Razor Page Applications</a></p>]]></content>
<summary type="html"><h3 id="問題"><a href="#問題" class="headerlink" title="問題"></a>問題</h3><p>ABP 預設會使用自己的驗證，如果要使用 Azure AD 驗證登入的話，<br>可以依 <a href="https://abp.io/c</summary>
<category term="Abp" scheme="https://rainmakerho.github.io/tags/Abp/"/>
<category term="AzureAD" scheme="https://rainmakerho.github.io/tags/AzureAD/"/>
<category term="Azure Active Directory" scheme="https://rainmakerho.github.io/tags/Azure-Active-Directory/"/>
<category term="OpenIdConnect" scheme="https://rainmakerho.github.io/tags/OpenIdConnect/"/>
</entry>
<entry>
<title>這個作業已經被取消，因為這個電腦受到限制</title>
<link href="https://rainmakerho.github.io/2025/01/15/windows-this-operation-has-been-cancelled-due-to-restrictions/"/>
<id>https://rainmakerho.github.io/2025/01/15/windows-this-operation-has-been-cancelled-due-to-restrictions/</id>
<published>2025-01-15T02:51:22.000Z</published>
<updated>2025-01-15T02:57:56.454Z</updated>
<content type="html"><![CDATA[<h3 id="問題"><a href="#問題" class="headerlink" title="問題"></a>問題</h3><p>開啟 Outlook 時，噴「Outlook 這個作業已經被取消，因為這個電腦受到限制。請連絡您的系統管理員」的錯誤。<br>但是用安全模式來開啟又是可以的<code>outlook.exe /safe</code> 。</p><h3 id="解決"><a href="#解決" class="headerlink" title="解決"></a>解決</h3><p>後來查到<a href="https://medium.com/@breich_84283/fix-this-operation-has-been-cancelled-due-to-restrictions-in-effect-on-this-computer-error-ec54825b30f">Fix “This operation has been cancelled due to restrictions in effect on this computer” Error</a>。<br>將機碼<br><code>\HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Policies\Explorer\</code><br>裡面的 <strong>DisallowRun</strong> Key ，刪除掉，就可以了。</p><ul><li>感謝 Terry 的回饋。</li></ul><h3 id="參考資源"><a href="#參考資源" class="headerlink" title="參考資源"></a>參考資源</h3><p><a href="https://medium.com/@breich_84283/fix-this-operation-has-been-cancelled-due-to-restrictions-in-effect-on-this-computer-error-ec54825b30f">Fix “This operation has been cancelled due to restrictions in effect on this computer” Error</a><br><a href="https://www.partitionwizard.com/clone-disk/this-operation-has-been-cancelled-due-to-restrictions.html">This Operation Has Been Cancelled Due to Restrictions?</a><br><a href="https://www.youtube.com/watch?v=rtCILYzvt4Y">This operation has been cancelled due to restrictions in effect on this computer</a></p>]]></content>
<summary type="html"><h3 id="問題"><a href="#問題" class="headerlink" title="問題"></a>問題</h3><p>開啟 Outlook 時，噴「Outlook 這個作業已經被取消，因為這個電腦受到限制。請連絡您的系統管理員」的錯誤。<br>但是用安全模式</summary>
<category term="Windows" scheme="https://rainmakerho.github.io/tags/Windows/"/>
<category term="Outlook" scheme="https://rainmakerho.github.io/tags/Outlook/"/>
<category term="operation has been cancelled" scheme="https://rainmakerho.github.io/tags/operation-has-been-cancelled/"/>
</entry>
<entry>
<title>ABP Dynamic Permissions，動態異動 ABP Permissions，不需重啟系統即可使用 Permissions</title>
<link href="https://rainmakerho.github.io/2025/01/02/abp-dynamic-add-permissions/"/>
<id>https://rainmakerho.github.io/2025/01/02/abp-dynamic-add-permissions/</id>
<published>2025-01-02T08:31:22.000Z</published>
<updated>2025-01-03T01:02:06.126Z</updated>
<content type="html"><![CDATA[<h3 id="問題"><a href="#問題" class="headerlink" title="問題"></a>問題</h3><p>預設 ABP 的權限會定義在繼承自<code>PermissionDefinitionProvider</code>的類別中，<br>當系統跑起來時，會自動將它們加到 Memory 之中，並存到<code>AbpPermissionGroups</code>及<code>AbpPermissions</code>資料表之中。<br>但是，如果要動態地去新增權限，要怎麼做呢?</p><h3 id="解法"><a href="#解法" class="headerlink" title="解法"></a>解法</h3><p>預設如果在<code>AbpPermissions</code> 資料表中的資料，卻沒定義在程式中，就不會被 Load 到 Memory 之中。<br>如果共用權限系統，那麼會變成需要將那些程式都各自加到各別系統中，而且只要有調整，就需要重新編譯程式!!!<br>所以，ABP Support dynamic permissions, 只要將設定<code>PermissionManagementOptions</code>的<code>IsDynamicPermissionStoreEnabled</code>屬性為<strong>true</strong>。<br>這樣就會連同資料庫中的 Permissions 資料一併加到 Memory 之中。<br>所以要如何動態新增權限呢?<br>可以參考<a href="https://github.com/abpframework/abp-samples/tree/master/DynamicPermission">ABP DynamicPermission Sample</a>的方式，<br>透過<code>IPermissionGroupDefinitionRecordRepository</code>及<code>IPermissionDefinitionRecordRepository</code>將權限加到資料庫，<br>再將 Cache 設定為過期，這樣就可以新增權限後，該權限就會在 Memory Cache 之中。</p><p>1.透過 ABP 建立 <a href="https://abp.io/docs/latest/tutorials/book-store/part-01?UI=MVC&DB=EF">BookStore Web 系統</a></p><p>2.加入<code>FixedLocalizableStringSerializer</code>避免沒設定多國語系而發生錯誤，</p><figure class="highlight csharp"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br></pre></td><td class="code"><pre><span class="line">[<span class="meta">Dependency(ReplaceServices = true)</span>]</span><br><span class="line">[<span class="meta">ExposeServices(typeof(LocalizableStringSerializer), typeof(ILocalizableStringSerializer), typeof(FixedLocalizableStringSerializer))</span>]</span><br><span class="line"><span class="keyword">public</span> <span class="keyword">class</span> <span class="title">FixedLocalizableStringSerializer</span> : <span class="title">LocalizableStringSerializer</span>, <span class="title">ITransientDependency</span></span><br><span class="line">{</span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="title">FixedLocalizableStringSerializer</span>(<span class="params">IOptions<AbpLocalizationOptions> localizationOptions</span>) : <span class="title">base</span>(<span class="params">localizationOptions</span>)</span></span><br><span class="line"> {</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">override</span> ILocalizableString <span class="title">Deserialize</span>(<span class="params"><span class="built_in">string</span>? <span class="keyword">value</span></span>)</span></span><br><span class="line"> {</span><br><span class="line"> <span class="keyword">if</span> (<span class="keyword">value</span> == <span class="literal">null</span>)</span><br><span class="line"> {</span><br><span class="line"> <span class="keyword">return</span> <span class="keyword">new</span> FixedLocalizableString(<span class="string">""</span>);</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">try</span></span><br><span class="line"> {</span><br><span class="line"> <span class="keyword">return</span> <span class="keyword">base</span>.Deserialize(<span class="keyword">value</span>);</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">catch</span></span><br><span class="line"> {</span><br><span class="line"> <span class="keyword">return</span> <span class="keyword">new</span> FixedLocalizableString(<span class="keyword">value</span>);</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>3.設定<strong>IsDynamicPermissionStoreEnabled</strong>為<strong>true</strong><br>在<code>BookStoreModule.cs</code>中<code>ConfigureServices</code>最後加入設定，</p><figure class="highlight csharp"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">Configure<PermissionManagementOptions>(options =></span><br><span class="line">{</span><br><span class="line"> options.IsDynamicPermissionStoreEnabled = <span class="literal">true</span>;</span><br><span class="line">});</span><br><span class="line">context.Services.Replace(ServiceDescriptor.Singleton<ILocalizableStringSerializer, FixedLocalizableStringSerializer>());</span><br><span class="line">Configure<Microsoft.AspNetCore.Mvc.JsonOptions>(options =></span><br><span class="line">{</span><br><span class="line"> options.JsonSerializerOptions.ReferenceHandler = ReferenceHandler.IgnoreCycles;</span><br><span class="line">});</span><br></pre></td></tr></table></figure><p>4.建立<code>PermissionDefinitionAppService.cs</code>來負責新增權限，<br>權限資料可以加入額外的一些屬性，例如 Url, Icon, Target ，可用在 Menu</p><figure class="highlight csharp"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">public</span> <span class="keyword">class</span> <span class="title">PermissionDefinitionAppService</span> : <span class="title">BookStoreAppService</span></span><br><span class="line">{</span><br><span class="line"></span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">readonly</span> IPermissionGroupDefinitionRecordRepository _permissionGroupRepository;</span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">readonly</span> IPermissionDefinitionRecordRepository _permissionRepository;</span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">readonly</span> IDistributedCache _distributedCache;</span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">readonly</span> DynamicPermissionDefinitionStore _dynamicPermissionDefinitionStore;</span><br><span class="line"> <span class="keyword">protected</span> <span class="keyword">readonly</span> AbpDistributedCacheOptions _cacheOptions;</span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">readonly</span> IDynamicPermissionDefinitionStoreInMemoryCache _cache;</span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">readonly</span> IPermissionDefinitionManager _permissionDefManager;</span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="title">PermissionDefinitionAppService</span>(<span class="params">IPermissionGroupDefinitionRecordRepository permissionGroupRepository,</span></span></span><br><span class="line"><span class="params"><span class="function"> IPermissionDefinitionRecordRepository permissionRepository,</span></span></span><br><span class="line"><span class="params"><span class="function"> IDistributedCache distributedCache,</span></span></span><br><span class="line"><span class="params"><span class="function"> DynamicPermissionDefinitionStore dynamicPermissionDefinitionStore,</span></span></span><br><span class="line"><span class="params"><span class="function"> IOptions<AbpDistributedCacheOptions> cacheOptions,</span></span></span><br><span class="line"><span class="params"><span class="function"> IDynamicPermissionDefinitionStoreInMemoryCache cache,</span></span></span><br><span class="line"><span class="params"><span class="function"> IPermissionDefinitionManager permissionDefManager</span>)</span></span><br><span class="line"> {</span><br><span class="line"> _permissionGroupRepository = permissionGroupRepository;</span><br><span class="line"> _permissionRepository = permissionRepository;</span><br><span class="line"> _distributedCache = distributedCache;</span><br><span class="line"> _dynamicPermissionDefinitionStore = dynamicPermissionDefinitionStore;</span><br><span class="line"> _cacheOptions = cacheOptions.Value;</span><br><span class="line"> _cache = cache;</span><br><span class="line"> _permissionDefManager = permissionDefManager;</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">async</span> Task <span class="title">PostPermissionAsync</span>(<span class="params"><span class="built_in">string</span> groupName, <span class="built_in">string</span> parentName, <span class="built_in">string</span> name, <span class="built_in">string</span> url, <span class="built_in">string</span> icon, <span class="built_in">string</span> target</span>)</span></span><br><span class="line"> {</span><br><span class="line"></span><br><span class="line"> <span class="keyword">var</span> permissionGroup = <span class="keyword">new</span> PermissionGroupDefinitionRecord</span><br><span class="line"> {</span><br><span class="line"> Name = groupName,</span><br><span class="line"> DisplayName = GetDisplayName(groupName)</span><br><span class="line"> };</span><br><span class="line"> <span class="keyword">var</span> <span class="keyword">group</span> = <span class="keyword">await</span> _permissionGroupRepository.GetListAsync();</span><br><span class="line"> <span class="keyword">if</span> (<span class="keyword">group</span>.All(x => x.Name != groupName))</span><br><span class="line"> {</span><br><span class="line"> <span class="keyword">await</span> _permissionGroupRepository.InsertAsync(permissionGroup);</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">var</span> permissions = <span class="keyword">await</span> _permissionRepository.GetListAsync();</span><br><span class="line"> <span class="keyword">if</span> (permissions.All(x => x.Name != name))</span><br><span class="line"> {</span><br><span class="line"> <span class="keyword">var</span> testPermission = <span class="keyword">new</span> PermissionDefinitionRecord</span><br><span class="line"> {</span><br><span class="line"> Name = name,</span><br><span class="line"> DisplayName = GetDisplayName(name),</span><br><span class="line"> GroupName = groupName,</span><br><span class="line"> IsEnabled = <span class="literal">true</span>,</span><br><span class="line"> ParentName = parentName,</span><br><span class="line"> MultiTenancySide = MultiTenancySides.Both</span><br><span class="line"> };</span><br><span class="line"> <span class="keyword">if</span>(!<span class="built_in">string</span>.IsNullOrEmpty(url))</span><br><span class="line"> {</span><br><span class="line"> testPermission.SetProperty(<span class="string">"Url"</span>, url);</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">if</span> (!<span class="built_in">string</span>.IsNullOrEmpty(icon))</span><br><span class="line"> {</span><br><span class="line"> testPermission.SetProperty(<span class="string">"Icon"</span>, icon);</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">if</span> (!<span class="built_in">string</span>.IsNullOrEmpty(target))</span><br><span class="line"> {</span><br><span class="line"> testPermission.SetProperty(<span class="string">"Target"</span>, target);</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">await</span> _permissionRepository.InsertAsync(testPermission);</span><br><span class="line"></span><br><span class="line"> <span class="keyword">await</span> _distributedCache.RemoveAsync(<span class="string">$"<span class="subst">{_cacheOptions.KeyPrefix}</span>_AbpInMemoryPermissionCacheStamp"</span>);</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">private</span> <span class="built_in">string</span> <span class="title">GetDisplayName</span>(<span class="params"><span class="built_in">string</span> name</span>)</span></span><br><span class="line"> {</span><br><span class="line"> <span class="comment">//L:BookStore,Permission:BookStore => 對應到語系檔內容</span></span><br><span class="line"> <span class="keyword">var</span> resourceName = <span class="string">"BookStore"</span>;</span><br><span class="line"> <span class="keyword">var</span> permissionName = <span class="string">$"Permission"</span>;</span><br><span class="line"> <span class="keyword">return</span> <span class="string">$"L:<span class="subst">{resourceName}</span>,<span class="subst">{permissionName}</span>:<span class="subst">{name}</span>"</span>;</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="title">Task</span><<span class="title">PermissionDefinition</span>> <span class="title">GetPermissionAsync</span>(<span class="params"><span class="built_in">string</span> name</span>)</span></span><br><span class="line"> {</span><br><span class="line"> <span class="keyword">return</span> _permissionDefManager.GetAsync(name);</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">private</span> <span class="keyword">static</span> LocalizableString <span class="title">L</span>(<span class="params"><span class="built_in">string</span> name</span>)</span></span><br><span class="line"> {</span><br><span class="line"> <span class="keyword">return</span> LocalizableString.Create<BookStoreResource>(name);</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>5.在 Page 中讓使用者設定權限資料，並呼叫<code>PermissionDefinitionAppService</code>來新增權限<br>Index.cshtml.cs</p><figure class="highlight csharp"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">public</span> <span class="keyword">class</span> <span class="title">IndexModel</span> : <span class="title">AbpPageModel</span></span><br><span class="line">{</span><br><span class="line"></span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">readonly</span> PermissionDefinition2AppService _permissionDefAppService;</span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="title">IndexModel</span>(<span class="params">PermissionDefinition2AppService permissionDefAppService</span>)</span></span><br><span class="line"> {</span><br><span class="line"> _permissionDefAppService = permissionDefAppService;</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> [<span class="meta">BindProperty</span>]</span><br><span class="line"> <span class="keyword">public</span> <span class="built_in">string</span> GroupName { <span class="keyword">get</span>; <span class="keyword">set</span>; }</span><br><span class="line"></span><br><span class="line"> [<span class="meta">BindProperty</span>]</span><br><span class="line"> <span class="keyword">public</span> <span class="built_in">string</span>? ParentPermissionName { <span class="keyword">get</span>; <span class="keyword">set</span>; }</span><br><span class="line"></span><br><span class="line"> [<span class="meta">BindProperty</span>]</span><br><span class="line"> <span class="keyword">public</span> <span class="built_in">string</span> PermissionName { <span class="keyword">get</span>; <span class="keyword">set</span>; }</span><br><span class="line"> [<span class="meta">BindProperty</span>]</span><br><span class="line"> <span class="keyword">public</span> <span class="built_in">string</span>? Icon { <span class="keyword">get</span>; <span class="keyword">set</span>; }</span><br><span class="line"></span><br><span class="line"> [<span class="meta">BindProperty</span>]</span><br><span class="line"> <span class="keyword">public</span> <span class="built_in">string</span>? Url { <span class="keyword">get</span>; <span class="keyword">set</span>; }</span><br><span class="line"></span><br><span class="line"> [<span class="meta">BindProperty</span>]</span><br><span class="line"> <span class="keyword">public</span> <span class="built_in">string</span>? Target { <span class="keyword">get</span>; <span class="keyword">set</span>; }</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">public</span> Task <span class="title">OnPostAddPermission</span>()</span></span><br><span class="line"> {</span><br><span class="line"> <span class="keyword">return</span> _permissionDefAppService.PostPermissionAsync(GroupName, ParentPermissionName, PermissionName, Url, Icon, Target);</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>Index.cshtml.cs</p><figure class="highlight html"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br></pre></td><td class="code"><pre><span class="line"><span class="tag"><<span class="name">div</span> <span class="attr">class</span>=<span class="string">"row"</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">div</span> <span class="attr">class</span>=<span class="string">"col-6"</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">div</span> <span class="attr">class</span>=<span class="string">"card"</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">div</span> <span class="attr">class</span>=<span class="string">"card-header"</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">div</span> <span class="attr">class</span>=<span class="string">"card-title text-body"</span>></span>Add Permissions<span class="tag"></<span class="name">div</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">div</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">div</span> <span class="attr">class</span>=<span class="string">"card-body pt-0"</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">form</span> <span class="attr">method</span>=<span class="string">"post"</span> <span class="attr">class</span>=<span class="string">"m-2"</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">div</span> <span class="attr">class</span>=<span class="string">"mb-3 col-12"</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">label</span> <span class="attr">for</span>=<span class="string">"formula"</span> <span class="attr">class</span>=<span class="string">"form-label"</span></span></span><br><span class="line"><span class="tag"> ></span>Group Name:</label</span><br><span class="line"> ></span><br><span class="line"> <span class="tag"><<span class="name">input</span></span></span><br><span class="line"><span class="tag"> <span class="attr">type</span>=<span class="string">"text"</span></span></span><br><span class="line"><span class="tag"> <span class="attr">class</span>=<span class="string">"form-control"</span></span></span><br><span class="line"><span class="tag"> <span class="attr">asp-for</span>=<span class="string">"GroupName"</span></span></span><br><span class="line"><span class="tag"> /></span></span><br><span class="line"> <span class="tag"></<span class="name">div</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">div</span> <span class="attr">class</span>=<span class="string">"mb-3 col-12"</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">label</span> <span class="attr">for</span>=<span class="string">"formula"</span> <span class="attr">class</span>=<span class="string">"form-label"</span></span></span><br><span class="line"><span class="tag"> ></span>Parent Permission Name:</label</span><br><span class="line"> ></span><br><span class="line"> <span class="tag"><<span class="name">input</span></span></span><br><span class="line"><span class="tag"> <span class="attr">type</span>=<span class="string">"text"</span></span></span><br><span class="line"><span class="tag"> <span class="attr">class</span>=<span class="string">"form-control"</span></span></span><br><span class="line"><span class="tag"> <span class="attr">asp-for</span>=<span class="string">"ParentPermissionName"</span></span></span><br><span class="line"><span class="tag"> /></span></span><br><span class="line"> <span class="tag"></<span class="name">div</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">div</span> <span class="attr">class</span>=<span class="string">"mb-3 col-12"</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">label</span> <span class="attr">for</span>=<span class="string">"formula"</span> <span class="attr">class</span>=<span class="string">"form-label"</span></span></span><br><span class="line"><span class="tag"> ></span>Permission Name:</label</span><br><span class="line"> ></span><br><span class="line"> <span class="tag"><<span class="name">input</span></span></span><br><span class="line"><span class="tag"> <span class="attr">type</span>=<span class="string">"text"</span></span></span><br><span class="line"><span class="tag"> <span class="attr">class</span>=<span class="string">"form-control"</span></span></span><br><span class="line"><span class="tag"> <span class="attr">asp-for</span>=<span class="string">"PermissionName"</span></span></span><br><span class="line"><span class="tag"> /></span></span><br><span class="line"> <span class="tag"></<span class="name">div</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">div</span> <span class="attr">class</span>=<span class="string">"mb-3 col-12"</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">label</span> <span class="attr">for</span>=<span class="string">"formula"</span> <span class="attr">class</span>=<span class="string">"form-label"</span>></span>Url:<span class="tag"></<span class="name">label</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">input</span> <span class="attr">type</span>=<span class="string">"text"</span> <span class="attr">class</span>=<span class="string">"form-control"</span> <span class="attr">asp-for</span>=<span class="string">"Url"</span> /></span></span><br><span class="line"> <span class="tag"></<span class="name">div</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">div</span> <span class="attr">class</span>=<span class="string">"mb-3 col-12"</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">label</span> <span class="attr">for</span>=<span class="string">"formula"</span> <span class="attr">class</span>=<span class="string">"form-label"</span>></span>Icon:<span class="tag"></<span class="name">label</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">input</span></span></span><br><span class="line"><span class="tag"> <span class="attr">type</span>=<span class="string">"text"</span></span></span><br><span class="line"><span class="tag"> <span class="attr">class</span>=<span class="string">"form-control"</span></span></span><br><span class="line"><span class="tag"> <span class="attr">asp-for</span>=<span class="string">"Icon"</span></span></span><br><span class="line"><span class="tag"> /></span></span><br><span class="line"> <span class="tag"></<span class="name">div</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">div</span> <span class="attr">class</span>=<span class="string">"mb-3 col-12"</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">label</span> <span class="attr">for</span>=<span class="string">"formula"</span> <span class="attr">class</span>=<span class="string">"form-label"</span>></span>Target:<span class="tag"></<span class="name">label</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">input</span></span></span><br><span class="line"><span class="tag"> <span class="attr">type</span>=<span class="string">"text"</span></span></span><br><span class="line"><span class="tag"> <span class="attr">class</span>=<span class="string">"form-control"</span></span></span><br><span class="line"><span class="tag"> <span class="attr">asp-for</span>=<span class="string">"Target"</span></span></span><br><span class="line"><span class="tag"> /></span></span><br><span class="line"> <span class="tag"></<span class="name">div</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">button</span></span></span><br><span class="line"><span class="tag"> <span class="attr">type</span>=<span class="string">"submit"</span></span></span><br><span class="line"><span class="tag"> <span class="attr">class</span>=<span class="string">"btn btn-success col-12"</span></span></span><br><span class="line"><span class="tag"> <span class="attr">asp-page-handler</span>=<span class="string">"AddPermission"</span></span></span><br><span class="line"><span class="tag"> ></span></span><br><span class="line"> 增加功能</span><br><span class="line"> <span class="tag"></<span class="name">button</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">form</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">div</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">div</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">div</span>></span></span><br><span class="line"><span class="tag"></<span class="name">div</span>></span></span><br></pre></td></tr></table></figure><p>6.在 <code>BookStoreMenuContributor.cs</code> 中，動態來依使用者的權限新增 Menu，</p><figure class="highlight csharp"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">public</span> <span class="keyword">class</span> <span class="title">BookStoreMenuContributor</span> : <span class="title">IMenuContributor</span></span><br><span class="line">{</span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">readonly</span> IServiceCollection _services;</span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">readonly</span> IPermissionChecker _permissionChecker;</span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="title">BookStoreMenuContributor</span>(<span class="params">IServiceCollection services</span>)</span></span><br><span class="line"> {</span><br><span class="line"> _services = services;</span><br><span class="line"> _permissionChecker = services.GetRequiredService<IPermissionChecker>();</span><br><span class="line"> }</span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">async</span> Task <span class="title">ConfigureMenuAsync</span>(<span class="params">MenuConfigurationContext context</span>)</span></span><br><span class="line"> {</span><br><span class="line"> <span class="keyword">if</span> (context.Menu.Name == StandardMenus.Main)</span><br><span class="line"> {</span><br><span class="line"> <span class="keyword">await</span> ConfigureMainMenuAsync(context);</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">private</span> <span class="keyword">async</span> Task <span class="title">ConfigureMainMenuAsync</span>(<span class="params"> MenuConfigurationContext context</span>)</span></span><br><span class="line"> {</span><br><span class="line"></span><br><span class="line"> <span class="keyword">var</span> administration = context.Menu.GetAdministration();</span><br><span class="line"> <span class="keyword">var</span> l = context.GetLocalizer<BookStoreResource>();</span><br><span class="line"></span><br><span class="line"> context.Menu.Items.Insert(</span><br><span class="line"> <span class="number">0</span>,</span><br><span class="line"> <span class="keyword">new</span> ApplicationMenuItem(</span><br><span class="line"> BookStoreMenus.Home,</span><br><span class="line"> l[<span class="string">"Menu:Home"</span>],</span><br><span class="line"> <span class="string">"~/"</span>,</span><br><span class="line"> icon: <span class="string">"fas fa-home"</span>,</span><br><span class="line"> order: <span class="number">0</span></span><br><span class="line"> )</span><br><span class="line"> );</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"> <span class="keyword">var</span> rootPermissionName = <span class="string">"BookStore.RootFun"</span>;</span><br><span class="line"> <span class="keyword">var</span> permissionDefManager = _services.GetRequiredService<IPermissionDefinitionManager>();</span><br><span class="line"> <span class="keyword">var</span> rootPermission = <span class="keyword">await</span> permissionDefManager.GetOrNullAsync(rootPermissionName);</span><br><span class="line"> <span class="keyword">var</span> isGrant = <span class="keyword">await</span> _permissionChecker.IsGrantedAsync(rootPermissionName);</span><br><span class="line"> <span class="keyword">if</span> (rootPermission != <span class="literal">null</span> && isGrant)</span><br><span class="line"> {</span><br><span class="line"> <span class="keyword">var</span> menu = <span class="keyword">await</span> ConfigureAPMenuAsync(context, rootPermission);</span><br><span class="line"> context.Menu.Items.Add(menu);</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">if</span> (BookStoreModule.IsMultiTenant)</span><br><span class="line"> {</span><br><span class="line"> administration.SetSubItemOrder(TenantManagementMenuNames.GroupName, <span class="number">1</span>);</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">else</span></span><br><span class="line"> {</span><br><span class="line"> administration.TryRemoveMenuItem(TenantManagementMenuNames.GroupName);</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">private</span> <span class="keyword">async</span> Task<ApplicationMenuItem> <span class="title">ConfigureAPMenuAsync</span>(<span class="params">MenuConfigurationContext context, PermissionDefinition permission</span>)</span></span><br><span class="line"> {</span><br><span class="line"> <span class="keyword">var</span> l = context.GetLocalizer<BookStoreResource>();</span><br><span class="line"> <span class="keyword">var</span> menu = <span class="keyword">new</span> ApplicationMenuItem(</span><br><span class="line"> permission.Name,</span><br><span class="line"> l[<span class="string">$"Permission:<span class="subst">{permission.Name}</span>"</span>]</span><br><span class="line"> );</span><br><span class="line"> <span class="keyword">if</span> (permission.Properties.ContainsKey(<span class="string">"Icon"</span>))</span><br><span class="line"> {</span><br><span class="line"> menu.Icon = permission.Properties[<span class="string">"Icon"</span>] <span class="keyword">as</span> <span class="built_in">string</span>;</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">if</span> (permission.Properties.ContainsKey(<span class="string">"Url"</span>))</span><br><span class="line"> {</span><br><span class="line"> menu.Url = permission.Properties[<span class="string">"Url"</span>] <span class="keyword">as</span> <span class="built_in">string</span>;</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">if</span> (permission.Properties.ContainsKey(<span class="string">"Target"</span>))</span><br><span class="line"> {</span><br><span class="line"> menu.Target = permission.Properties[<span class="string">"Target"</span>] <span class="keyword">as</span> <span class="built_in">string</span>;</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">foreach</span> (<span class="keyword">var</span> child <span class="keyword">in</span> permission.Children)</span><br><span class="line"> {</span><br><span class="line"> <span class="keyword">var</span> grant = <span class="keyword">await</span> _permissionChecker.IsGrantedAsync(child.Name);</span><br><span class="line"> <span class="keyword">if</span> (grant)</span><br><span class="line"> {</span><br><span class="line"> <span class="keyword">var</span> childMenu = <span class="keyword">await</span> ConfigureAPMenuAsync(context, child);</span><br><span class="line"> menu.Items.Add(childMenu);</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">return</span> menu;</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>使用 admin 登入後，動態加入權限</p><img src="/2025/01/02/abp-dynamic-add-permissions/01.png" class="" title="加入權限"><p>在設定權限功能，可以看到上一步新增的權限，已經可以看到並設定，</p><img src="/2025/01/02/abp-dynamic-add-permissions/02.png" class="" title="加入權限"><p>重整畫面，就可以依權限來呈現 Menu。</p><h3 id="參考資源"><a href="#參考資源" class="headerlink" title="參考資源"></a>參考資源</h3><p><a href="https://github.com/abpframework/abp/discussions/21779">How to Dynamic Add/Update Permissions and Refresh Cache in ABP</a><br><a href="https://github.com/abpframework/abp-samples/tree/master/DynamicPermission">ABP DynamicPermission Sample</a><br><a href="https://abp.io/support/questions/5671/Generate-Permission-Automatically-from-external-service">Generate Permission Automatically from external service</a></p>]]></content>
<summary type="html"><h3 id="問題"><a href="#問題" class="headerlink" title="問題"></a>問題</h3><p>預設 ABP 的權限會定義在繼承自<code>PermissionDefinitionProvider</code>的類別中，<br>當系統</summary>
<category term="ABP" scheme="https://rainmakerho.github.io/tags/ABP/"/>
<category term="Dynamic" scheme="https://rainmakerho.github.io/tags/Dynamic/"/>
<category term="Permission" scheme="https://rainmakerho.github.io/tags/Permission/"/>
<category term="PermissionDefinitionProvider" scheme="https://rainmakerho.github.io/tags/PermissionDefinitionProvider/"/>
<category term="AbpPermissions" scheme="https://rainmakerho.github.io/tags/AbpPermissions/"/>
<category term="AbpPermissionGroups" scheme="https://rainmakerho.github.io/tags/AbpPermissionGroups/"/>
<category term="IsDynamicPermissionStoreEnabled" scheme="https://rainmakerho.github.io/tags/IsDynamicPermissionStoreEnabled/"/>
</entry>
<entry>
<title>ABP A possible object cycle was detected.</title>
<link href="https://rainmakerho.github.io/2024/12/31/abp-json-possible-object-cycle-was-detected-err/"/>
<id>https://rainmakerho.github.io/2024/12/31/abp-json-possible-object-cycle-was-detected-err/</id>
<published>2024-12-31T06:59:03.000Z</published>
<updated>2024-12-31T07:11:32.354Z</updated>
<content type="html"><![CDATA[<h3 id="問題"><a href="#問題" class="headerlink" title="問題"></a>問題</h3><p>在 Abp AppService 中回傳的物件如果有 recursion 的狀況，像是取回 <code>PermissionGroupDefinition</code> ，會包含 <code>List<PermissionDefinition></code> ，而 <code>PermissionDefinition</code> 又會包含 <code>List<PermissionDefinition></code>，所以就會出現以下的錯誤，</p><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">[ERR] A possible object cycle was detected. This can either be due to a cycle or if the object depth is larger than the maximum allowed depth of 32. Consider using ReferenceHandler.Preserve on JsonSerializerOptions to support cycles. Path: $.Permissions.Children.Parent.Children.Parent.Children.Parent.Children.Parent.Children.Parent.Children.Parent.Children.Parent.Children.Parent.Children.Parent.Children.</span><br><span class="line">System.Text.Json.JsonException: A possible object cycle was detected. This can either be due to a cycle or if the object depth is larger than the maximum allowed depth of 32. Consider using ReferenceHandler.Preserve on JsonSerializerOptions to support cycles. Path: $.Permissions.Children.Parent.Children.Parent.Children.Parent.Children.Parent.Children.Parent.Children.Parent.Children.Parent.Children.Parent.Children.Parent.Children.</span><br><span class="line"> at System.Text.Json.ThrowHelper.ThrowJsonException_SerializerCycleDetected(Int32 maxDepth)</span><br><span class="line"> at System.Text.Json.Serialization.JsonConverter`1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state)</span><br></pre></td></tr></table></figure><h3 id="解法"><a href="#解法" class="headerlink" title="解法"></a>解法</h3><p>參考<a href="https://github.com/abpframework/abp/issues/10591#issuecomment-1370050436">Adding configuration for JSON serializer providers doesn’t work</a>，在 Web 專案的 Module 檔案中，加入以下的設定就可以正常運作。</p><figure class="highlight csharp"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">Configure<Microsoft.AspNetCore.Mvc.JsonOptions>(options =></span><br><span class="line">{</span><br><span class="line"> options.JsonSerializerOptions.ReferenceHandler = ReferenceHandler.IgnoreCycles;</span><br><span class="line">});</span><br></pre></td></tr></table></figure><h3 id="參考資源"><a href="#參考資源" class="headerlink" title="參考資源"></a>參考資源</h3><p><a href="https://github.com/abpframework/abp/issues/10591#issuecomment-1370050436">Adding configuration for JSON serializer providers doesn’t work</a></p>]]></content>
<summary type="html"><h3 id="問題"><a href="#問題" class="headerlink" title="問題"></a>問題</h3><p>在 Abp AppService 中回傳的物件如果有 recursion 的狀況，像是取回 <code>PermissionGroupDef</summary>
<category term="json" scheme="https://rainmakerho.github.io/tags/json/"/>
<category term="JsonSerializerOptions" scheme="https://rainmakerho.github.io/tags/JsonSerializerOptions/"/>
<category term="abp" scheme="https://rainmakerho.github.io/tags/abp/"/>
<category term="ReferenceHandler.Preserve" scheme="https://rainmakerho.github.io/tags/ReferenceHandler-Preserve/"/>
<category term="recursion" scheme="https://rainmakerho.github.io/tags/recursion/"/>
</entry>
<entry>
<title>使用 Vanna 來練習 Text2SQL</title>
<link href="https://rainmakerho.github.io/2024/12/03/text2sql-nl2sql-use-vanna-ai/"/>
<id>https://rainmakerho.github.io/2024/12/03/text2sql-nl2sql-use-vanna-ai/</id>
<published>2024-12-03T05:27:09.000Z</published>
<updated>2024-12-03T06:42:24.332Z</updated>
<content type="html"><![CDATA[<h3 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h3><p><a href="https://github.com/vanna-ai/vanna">Vanna</a>可以將 Table Schema, 文件說明, 問題/SQL 存到向量資料庫之中。當使用者提出問題時，會以取出相關的 Table 資訊，結合 Prompt 去詢問 LLM 來產生 SQL 。<br>再執行 SQL ，並以不同格式來呈現結果。<br>以下就以 MSSQL 的 AdventureWorks2022 DB 來練習。<br>環境: Windows 10, VS Code, chromadb, openai, SQL Server</p><h3 id="練習"><a href="#練習" class="headerlink" title="練習"></a>練習</h3><p>1.建立 text2sql 目錄</p><p>2.透過 VS Code 開啟 text2sql 目錄</p><p>3.建立 <code>vannasql.ipynb</code> ，選擇 python</p><p>4.安裝 vanna 並包含 chromadb, openai</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">%pip install vanna[chromadb,openai]</span><br></pre></td></tr></table></figure><ul><li>註: 如果沒安裝過 pyodbc ，也請安裝它 <code>%pip install pyodbc</code></li></ul><p>5.建立 vanna instance</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">class</span> <span class="title class_">MyVanna</span>(ChromaDB_VectorStore, OpenAI_Chat):</span><br><span class="line"> <span class="keyword">def</span> <span class="title function_">__init__</span>(<span class="params">self, config=<span class="literal">None</span></span>):</span><br><span class="line"> ChromaDB_VectorStore.__init__(self, config=config)</span><br><span class="line"> OpenAI_Chat.__init__(self, config=config)</span><br><span class="line">apikey = <span class="string">'openai apikey'</span>;</span><br><span class="line">vn = MyVanna(config={<span class="string">'api_key'</span>: apikey, <span class="string">'model'</span>: <span class="string">'gpt-4o'</span>})</span><br></pre></td></tr></table></figure><p>6.連接 db</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">vn.connect_to_mssql(odbc_conn_str=<span class="string">'DRIVER={ODBC Driver 17 for SQL Server};SERVER=.;DATABASE=AdventureWorks2022;Trusted_Connection=yes;'</span>)</span><br></pre></td></tr></table></figure><p>7.取出 Table Schema，建立訓練資料</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">df_information_schema = vn.run_sql(<span class="string">"SELECT * FROM INFORMATION_SCHEMA.COLUMNS"</span>)</span><br><span class="line">plan = vn.get_training_plan_generic(df_information_schema)</span><br><span class="line"><span class="comment">#plan</span></span><br><span class="line">vn.train(plan=plan)</span><br></pre></td></tr></table></figure><ul><li>註:在目錄中，它會產生<code>chroma.sqlite3</code>及許多的<code>Guid</code>的目錄，就是 Vanna 的訓練資料，<br>如果要重新訓練資料，就將這些目錄及檔案刪除掉。</li></ul><p>8.詢問問題<br>訓練好之後，就可以開始詢問它，如下，</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">vn.ask(<span class="string">'Find out the top 5 best salespeople'</span>)</span><br></pre></td></tr></table></figure><img src="/2024/12/03/text2sql-nl2sql-use-vanna-ai/01.png" class="" title="vn.ask"><p>9.執行 vanna.flask 來建立 Vanna Web 網站</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> vanna.flask <span class="keyword">import</span> VannaFlaskApp</span><br><span class="line"></span><br><span class="line">app = VannaFlaskApp(vn)</span><br><span class="line">app.run()</span><br></pre></td></tr></table></figure><img src="/2024/12/03/text2sql-nl2sql-use-vanna-ai/02.png" class="" title="vanna.flask"><p>當網站建立起來後，可以在網頁中進行查詢。<br>如果查詢結果有錯誤，可以進行手動修正。</p><img src="/2024/12/03/text2sql-nl2sql-use-vanna-ai/03.png" class="" title="vanna webUI"><p>每個 Table 它會建立的訓練資料如下，</p><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">The following columns are in the SalesTaxRate table in the AdventureWorks2022 database:</span><br><span class="line">| | TABLE_CATALOG | TABLE_SCHEMA | TABLE_NAME | COLUMN_NAME | DATA_TYPE | |---:|:-------------------|:---------------|:-------------|:----------------|:-----------------| | 0 | AdventureWorks2022 | Sales | SalesTaxRate | SalesTaxRateID | int | | 1 | AdventureWorks2022 | Sales | SalesTaxRate | StateProvinceID | int | | 2 | AdventureWorks2022 | Sales | SalesTaxRate | TaxType | tinyint | | 3 | AdventureWorks2022 | Sales | SalesTaxRate | TaxRate | smallmoney | | 4 | AdventureWorks2022 | Sales | SalesTaxRate | Name | nvarchar | | 5 | AdventureWorks2022 | Sales | SalesTaxRate | rowguid | uniqueidentifier | | 6 | AdventureWorks2022 | Sales | SalesTaxRate | ModifiedDate | datetime |</span><br></pre></td></tr></table></figure><h3 id="參考資源"><a href="#參考資源" class="headerlink" title="參考資源"></a>參考資源</h3><p><a href="https://vanna.ai/docs/mssql-openai-standard-chromadb/">Generating SQL for Microsoft SQL Server using OpenAI, ChromaDB</a><br><a href="https://github.com/vanna-ai/vanna">Vanna</a><br><a href="https://blog.csdn.net/qq_49070564/article/details/139717376">AI 与数据库交互</a></p>]]></content>
<summary type="html"><h3 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h3><p><a href="https://github.com/vanna-ai/vanna">Vanna</a>可以將 Table Schema,</summary>
<category term="mssql" scheme="https://rainmakerho.github.io/tags/mssql/"/>
<category term="Text2SQL" scheme="https://rainmakerho.github.io/tags/Text2SQL/"/>
<category term="NL2SQL" scheme="https://rainmakerho.github.io/tags/NL2SQL/"/>
<category term="vanna-ai" scheme="https://rainmakerho.github.io/tags/vanna-ai/"/>
</entry>
<entry>
<title>判斷字串是否包含簡體中文(是否為繁體中文)</title>
<link href="https://rainmakerho.github.io/2024/12/02/determine-string-is-simplified-chinese-hans/"/>
<id>https://rainmakerho.github.io/2024/12/02/determine-string-is-simplified-chinese-hans/</id>
<published>2024-12-02T01:04:38.000Z</published>
<updated>2024-12-02T01:50:51.974Z</updated>
<content type="html"><![CDATA[<h3 id="問題"><a href="#問題" class="headerlink" title="問題"></a>問題</h3><p>最近有個需求是要判斷使用者輸入的字串是否有<strong>簡體中文</strong>，<br>不好透過字串的編碼範圍來判斷。<br>所以，將使用者的輸入轉成<strong>繁體中文</strong>，<br>如果相同，則表示使用者輸入的是<strong>繁體中文</strong>。<br>如果不相同，則表示使用者輸入的<strong>不是繁體中文</strong>。</p><h3 id="實作"><a href="#實作" class="headerlink" title="實作"></a>實作</h3><p>所以可以透過<strong>OpenCCNET</strong>這個套件來幫我們進行<strong>簡/繁</strong>轉換。</p><p>1.加入 OpenCCNET Nuget 套件</p><p>2.進行轉換測試，</p><figure class="highlight csharp"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"><span class="doctag">///</span> <span class="doctag"><summary></span></span></span><br><span class="line"><span class="comment"><span class="doctag">///</span> 判斷字串是否為繁體中文</span></span><br><span class="line"><span class="comment"><span class="doctag">///</span> <span class="doctag"></summary></span></span></span><br><span class="line"><span class="comment"><span class="doctag">///</span> <span class="doctag"><param name="s"></span><span class="doctag"></param></span></span></span><br><span class="line"><span class="comment"><span class="doctag">///</span> <span class="doctag"><returns></span><span class="doctag"></returns></span></span></span><br><span class="line"><span class="function"><span class="keyword">private</span> <span class="keyword">static</span> (<span class="params"><span class="built_in">bool</span>, List<<span class="built_in">string</span>></span>) <span class="title">CheckHant</span>(<span class="params"><span class="built_in">string</span> s</span>)</span></span><br><span class="line">{</span><br><span class="line"> <span class="keyword">var</span> result = <span class="keyword">new</span> List<<span class="built_in">string</span>>();</span><br><span class="line"> <span class="keyword">var</span> strHant = ZhConverter.HansToHant(s);</span><br><span class="line"> <span class="keyword">var</span> isTW = (strHant == s);</span><br><span class="line"> <span class="keyword">foreach</span> (<span class="built_in">char</span> c <span class="keyword">in</span> s)</span><br><span class="line"> {</span><br><span class="line"> <span class="keyword">var</span> cn = c.ToString();</span><br><span class="line"> <span class="keyword">var</span> tw = ZhConverter.HansToHant(cn);</span><br><span class="line"> <span class="keyword">if</span>(tw != cn)</span><br><span class="line"> {</span><br><span class="line"> result.Add(cn);</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">return</span> (isTW, result);</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">var</span> s1 = <span class="string">"這是繁體中文abc"</span>;</span><br><span class="line"><span class="keyword">var</span> s2 = <span class="string">"这是繁体中文abc"</span>;</span><br><span class="line"><span class="keyword">var</span> s3 = <span class="string">"これは繁体字中国語のabcです"</span>;</span><br><span class="line">ZhConverter.Initialize();</span><br><span class="line"><span class="keyword">var</span> (isS1TW, s1CnResult) = CheckHant(s1);</span><br><span class="line">Console.WriteLine(<span class="string">$"<span class="subst">{s1}</span>是繁體中文嗎?<span class="subst">{isS1TW}</span>"</span>);</span><br><span class="line"><span class="keyword">var</span> (isS2TW, s2CnResult) = CheckHant(s2);</span><br><span class="line">Console.WriteLine(<span class="string">$"<span class="subst">{s2}</span>是繁體中文嗎?<span class="subst">{isS2TW}</span>"</span>);</span><br><span class="line"><span class="keyword">var</span> (isS3TW, s3CnResult) = CheckHant(s3);</span><br><span class="line">Console.WriteLine(<span class="string">$"<span class="subst">{s3}</span>是繁體中文嗎?<span class="subst">{isS3TW}</span>"</span>);</span><br><span class="line"></span><br></pre></td></tr></table></figure><h3 id="參考資訊"><a href="#參考資訊" class="headerlink" title="參考資訊"></a>參考資訊</h3><p><a href="https://github.com/CosineG/OpenCC.NET">OpenCCNET</a></p>]]></content>
<summary type="html"><h3 id="問題"><a href="#問題" class="headerlink" title="問題"></a>問題</h3><p>最近有個需求是要判斷使用者輸入的字串是否有<strong>簡體中文</strong>，<br>不好透過字串的編碼範圍來判斷。<br>所以，將</summary>
<category term=".NET" scheme="https://rainmakerho.github.io/tags/NET/"/>
<category term="zh-Hans" scheme="https://rainmakerho.github.io/tags/zh-Hans/"/>
<category term="zh-Hant" scheme="https://rainmakerho.github.io/tags/zh-Hant/"/>
<category term="zh-CN" scheme="https://rainmakerho.github.io/tags/zh-CN/"/>
<category term="zh-TW" scheme="https://rainmakerho.github.io/tags/zh-TW/"/>
<category term="OpenCCNET" scheme="https://rainmakerho.github.io/tags/OpenCCNET/"/>
</entry>
<entry>
<title>Windows 工作排程器中的排程忽然不跑了?</title>
<link href="https://rainmakerho.github.io/2024/11/29/windows-task-dont-work/"/>
<id>https://rainmakerho.github.io/2024/11/29/windows-task-dont-work/</id>
<published>2024-11-29T05:26:31.000Z</published>
<updated>2024-11-29T05:42:46.746Z</updated>
<content type="html"><![CDATA[<h3 id="問題"><a href="#問題" class="headerlink" title="問題"></a>問題</h3><p>今天同事詢問客戶端將工作排程器中的一個排程更換執行人員後，<br>設定每天週期執行、每天隔 5 分去跑一次。<br>但是，設定完成後，一直都沒有跑。</p><h3 id="解法"><a href="#解法" class="headerlink" title="解法"></a>解法</h3><p>建立一個新的 Task ，並先跑簡單的 Bat 檔案看能不能跑。<br>結果是可以正常執行。<br>再直接去跑 EXE ，也是可以正常執行的。</p><p>於是將有問題的 Task 及沒問題的 Task 匯出 XML 來比較，<br>結果發現有問題的 Task 中，沒有<code><Duration>P1D</Duration></code>這個內容。</p><p>也就是有問題的排程設定每 5 分執行，但<strong>持續時間為(F):<strong>這個欄位是設定成</strong>不限制</strong><br>而沒問題的排程設定中，<strong>持續時間為(F):<strong>這個欄位是設定成</strong>1 天</strong></p><img src="/2024/11/29/windows-task-dont-work/01.png" class="" title="排程設定"><p>將<strong>持續時間為(F):<strong>這個欄位，從</strong>不限制</strong>改成<strong>1 天</strong>，<br>就可以正常執行了。</p>]]></content>
<summary type="html"><h3 id="問題"><a href="#問題" class="headerlink" title="問題"></a>問題</h3><p>今天同事詢問客戶端將工作排程器中的一個排程更換執行人員後，<br>設定每天週期執行、每天隔 5 分去跑一次。<br>但是，設定完成後，一直都</summary>
<category term="Windows" scheme="https://rainmakerho.github.io/tags/Windows/"/>
<category term="Task" scheme="https://rainmakerho.github.io/tags/Task/"/>
<category term="工作排程器" scheme="https://rainmakerho.github.io/tags/%E5%B7%A5%E4%BD%9C%E6%8E%92%E7%A8%8B%E5%99%A8/"/>
<category term="Duration" scheme="https://rainmakerho.github.io/tags/Duration/"/>
</entry>
<entry>
<title>abp AppService 覆寫 Method 的 Authorize?</title>
<link href="https://rainmakerho.github.io/2024/11/08/aspnet-authorize-policy-override/"/>
<id>https://rainmakerho.github.io/2024/11/08/aspnet-authorize-policy-override/</id>
<published>2024-11-08T01:21:22.000Z</published>
<updated>2024-11-08T03:08:24.852Z</updated>
<content type="html"><![CDATA[<h3 id="問題"><a href="#問題" class="headerlink" title="問題"></a>問題</h3><p>同事繼承 Abp 範例中的 AuthorAppService ，將 UpdateAsync 設定成 virtual ，允許子類別去覆寫。<br>子類別在覆寫的 <code>UpdateAsync</code> 上，加上另個 Authorize 的設定，<br>結果要執行<code>UpdateAsync</code>變成需要有 2 個 Policy 的權限才可以執行。</p><h3 id="測試"><a href="#測試" class="headerlink" title="測試"></a>測試</h3><p>範例的<code>AuthorAppService</code>，其中的<code>UpdateAsync</code>設定成<code>virtual</code>，如下，</p><figure class="highlight csharp"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br></pre></td><td class="code"><pre><span class="line">[<span class="meta">Authorize(BookStorePermissions.Authors.Default)</span>]</span><br><span class="line"><span class="keyword">public</span> <span class="keyword">class</span> <span class="title">AuthorAppService</span> : <span class="title">BookStoreAppService</span>, <span class="title">IAuthorAppService</span></span><br><span class="line">{</span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">readonly</span> IAuthorRepository _authorRepository;</span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">readonly</span> AuthorManager _authorManager;</span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="title">AuthorAppService</span>(<span class="params"></span></span></span><br><span class="line"><span class="params"><span class="function"> IAuthorRepository authorRepository,</span></span></span><br><span class="line"><span class="params"><span class="function"> AuthorManager authorManager</span>)</span></span><br><span class="line"> {</span><br><span class="line"> _authorRepository = authorRepository;</span><br><span class="line"> _authorManager = authorManager;</span><br><span class="line"> }</span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">async</span> Task<AuthorDto> <span class="title">GetAsync</span>(<span class="params">Guid id</span>)</span></span><br><span class="line"> {</span><br><span class="line"> <span class="keyword">var</span> author = <span class="keyword">await</span> _authorRepository.GetAsync(id);</span><br><span class="line"> <span class="keyword">return</span> ObjectMapper.Map<Author, AuthorDto>(author);</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="keyword">public</span> <span class="keyword">async</span> Task<PagedResultDto<AuthorDto>> GetListAsync(GetAuthorListDto input)</span><br><span class="line"> {</span><br><span class="line"> <span class="keyword">if</span>(input.Sorting.IsNullOrWhiteSpace())</span><br><span class="line"> {</span><br><span class="line"> input.Sorting = <span class="keyword">nameof</span>(Author.Name);</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="keyword">var</span> authors = <span class="keyword">await</span> _authorRepository.GetListAsync(</span><br><span class="line"> input.SkipCount,</span><br><span class="line"> input.MaxResultCount,</span><br><span class="line"> input.Sorting,</span><br><span class="line"> input.Filter</span><br><span class="line"> );</span><br><span class="line"></span><br><span class="line"> <span class="keyword">var</span> totalCount = input.Filter == <span class="literal">null</span></span><br><span class="line"> ? <span class="keyword">await</span> _authorRepository.CountAsync()</span><br><span class="line"> : <span class="keyword">await</span> _authorRepository.CountAsync(</span><br><span class="line"> author => author.Name.Contains(input.Filter)</span><br><span class="line"> );</span><br><span class="line"></span><br><span class="line"> <span class="keyword">return</span> <span class="keyword">new</span> PagedResultDto<AuthorDto>(</span><br><span class="line"> totalCount,</span><br><span class="line"> ObjectMapper.Map<List<Author>, List<AuthorDto>>(authors)</span><br><span class="line"> );</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> [<span class="meta">Authorize(BookStorePermissions.Authors.Create)</span>]</span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">async</span> Task<AuthorDto> <span class="title">CreateAsync</span>(<span class="params">CreateAuthorDto input</span>)</span></span><br><span class="line"> {</span><br><span class="line"> <span class="keyword">var</span> author = <span class="keyword">await</span> _authorManager.CreateAsync(</span><br><span class="line"> input.Name,</span><br><span class="line"> input.BirthDate,</span><br><span class="line"> input.ShortBio</span><br><span class="line"> );</span><br><span class="line"></span><br><span class="line"> <span class="keyword">await</span> _authorRepository.InsertAsync(author);</span><br><span class="line"></span><br><span class="line"> <span class="keyword">return</span> ObjectMapper.Map<Author, AuthorDto>(author);</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"> [<span class="meta">Authorize(BookStorePermissions.Authors.Delete)</span>]</span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">async</span> Task <span class="title">DeleteAsync</span>(<span class="params">Guid id</span>)</span></span><br><span class="line"> {</span><br><span class="line"> <span class="keyword">await</span> _authorRepository.DeleteAsync(id);</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"> [<span class="meta">Authorize(BookStorePermissions.Authors.Edit)</span>]</span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">virtual</span> <span class="keyword">async</span> Task <span class="title">UpdateAsync</span>(<span class="params">Guid id, UpdateAuthorDto input</span>)</span></span><br><span class="line"> {</span><br><span class="line"> <span class="keyword">var</span> author = <span class="keyword">await</span> _authorRepository.GetAsync(id);</span><br><span class="line"> <span class="keyword">if</span>(author.Name != input.Name)</span><br><span class="line"> {</span><br><span class="line"> <span class="keyword">await</span> _authorManager.ChangeNameAsync(author, input.Name);</span><br><span class="line"> }</span><br><span class="line"> author.BirthDate = input.BirthDate;</span><br><span class="line"> author.ShortBio = input.ShortBio;</span><br><span class="line"></span><br><span class="line"> <span class="keyword">await</span> _authorRepository.UpdateAsync(author);</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>因為要多加個<code>BookStorePermissions.Authors.Edit2</code> Permission，<br>所以要在 <code>BookStorePermissions中的Authors Class</code> 中加入</p><figure class="highlight csharp"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">class</span> <span class="title">BookStorePermissions</span></span><br><span class="line">{</span><br><span class="line"> <span class="keyword">public</span> <span class="keyword">const</span> <span class="built_in">string</span> GroupName = <span class="string">"BookStore"</span>;</span><br><span class="line"> <span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">class</span> <span class="title">Books</span></span><br><span class="line"> {</span><br><span class="line"> <span class="keyword">public</span> <span class="keyword">const</span> <span class="built_in">string</span> Default = GroupName + <span class="string">".Books"</span>;</span><br><span class="line"> <span class="keyword">public</span> <span class="keyword">const</span> <span class="built_in">string</span> Create = Default + <span class="string">".Create"</span>;</span><br><span class="line"> <span class="keyword">public</span> <span class="keyword">const</span> <span class="built_in">string</span> Edit = Default + <span class="string">".Edit"</span>;</span><br><span class="line"> <span class="keyword">public</span> <span class="keyword">const</span> <span class="built_in">string</span> Delete = Default + <span class="string">".Delete"</span>;</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">class</span> <span class="title">Authors</span></span><br><span class="line"> {</span><br><span class="line"> <span class="keyword">public</span> <span class="keyword">const</span> <span class="built_in">string</span> Default = GroupName + <span class="string">".Authors"</span>;</span><br><span class="line"> <span class="keyword">public</span> <span class="keyword">const</span> <span class="built_in">string</span> Create = Default + <span class="string">".Create"</span>;</span><br><span class="line"> <span class="keyword">public</span> <span class="keyword">const</span> <span class="built_in">string</span> Edit = Default + <span class="string">".Edit"</span>;</span><br><span class="line"> <span class="keyword">public</span> <span class="keyword">const</span> <span class="built_in">string</span> Edit2 = Default + <span class="string">".Edit2"</span>;</span><br><span class="line"> <span class="keyword">public</span> <span class="keyword">const</span> <span class="built_in">string</span> Delete = Default + <span class="string">".Delete"</span>;</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>在<code>BookStorePermissionDefinitionProvider</code>中也要加入<code>Edit2</code>，<br><code>authorsPermission.AddChild(BookStorePermissions.Authors.Edit2, L("Permission:Authors.Edit"));</code>，如下，</p><figure class="highlight csharp"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">public</span> <span class="keyword">class</span> <span class="title">BookStorePermissionDefinitionProvider</span> : <span class="title">PermissionDefinitionProvider</span></span><br><span class="line">{</span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">override</span> <span class="keyword">void</span> <span class="title">Define</span>(<span class="params">IPermissionDefinitionContext context</span>)</span></span><br><span class="line"> {</span><br><span class="line"> <span class="keyword">var</span> bookStoreGroup = context.AddGroup(BookStorePermissions.GroupName, L(<span class="string">"Permission:BookStore"</span>));</span><br><span class="line"></span><br><span class="line"> <span class="keyword">var</span> booksPermission = bookStoreGroup.AddPermission(BookStorePermissions.Books.Default, L(<span class="string">"Permission:Books"</span>));</span><br><span class="line"> booksPermission.AddChild(BookStorePermissions.Books.Create, L(<span class="string">"Permission:Books.Create"</span>));</span><br><span class="line"> booksPermission.AddChild(BookStorePermissions.Books.Edit, L(<span class="string">"Permission:Books.Edit"</span>));</span><br><span class="line"> booksPermission.AddChild(BookStorePermissions.Books.Delete, L(<span class="string">"Permission:Books.Delete"</span>));</span><br><span class="line"></span><br><span class="line"> <span class="keyword">var</span> authorsPermission = bookStoreGroup.AddPermission(BookStorePermissions.Authors.Default, L(<span class="string">"Permission:Authors"</span>));</span><br><span class="line"> authorsPermission.AddChild(BookStorePermissions.Authors.Create, L(<span class="string">"Permission:Authors.Create"</span>));</span><br><span class="line"> authorsPermission.AddChild(BookStorePermissions.Authors.Edit, L(<span class="string">"Permission:Authors.Edit"</span>));</span><br><span class="line"> authorsPermission.AddChild(BookStorePermissions.Authors.Delete, L(<span class="string">"Permission:Authors.Delete"</span>));</span><br><span class="line"> authorsPermission.AddChild(BookStorePermissions.Authors.Edit2, L(<span class="string">"Permission:Authors.Edit"</span>));</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">private</span> <span class="keyword">static</span> LocalizableString <span class="title">L</span>(<span class="params"><span class="built_in">string</span> name</span>)</span></span><br><span class="line"> {</span><br><span class="line"> <span class="keyword">return</span> LocalizableString.Create<BookStoreResource>(name);</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>接著實作要繼承<code>AuthorAppService</code>的子類別<code>CustomAuthorAppService</code>，</p><figure class="highlight csharp"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><span class="line">[<span class="meta">Dependency(ReplaceServices = true)</span>]</span><br><span class="line">[<span class="meta">ExposeServices(typeof(AuthorAppService))</span>]</span><br><span class="line"><span class="keyword">public</span> <span class="keyword">class</span> <span class="title">CustomAuthorAppService</span> : <span class="title">AuthorAppService</span></span><br><span class="line">{</span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="title">CustomAuthorAppService</span>(<span class="params"></span></span></span><br><span class="line"><span class="params"><span class="function"> IAuthorRepository authorRepository,</span></span></span><br><span class="line"><span class="params"><span class="function"> AuthorManager authorManager</span>)</span></span><br><span class="line"><span class="function"> : <span class="title">base</span>(<span class="params">authorRepository, authorManager</span>)</span></span><br><span class="line"> {</span><br><span class="line"></span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> [<span class="meta">Authorize(BookStorePermissions.Authors.Edit2)</span>]</span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">override</span> <span class="keyword">async</span> Task <span class="title">UpdateAsync</span>(<span class="params">Guid id, UpdateAuthorDto input</span>)</span></span><br><span class="line"> {</span><br><span class="line"> <span class="keyword">await</span> <span class="keyword">base</span>.UpdateAsync(id, input);</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>新增一個<code>MyPermissionChecker</code>來查看執行<code>UpdateAsync</code>會需要那些的 Policy，</p><figure class="highlight csharp"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">public</span> <span class="keyword">class</span> <span class="title">MyPermissionChecker</span> : <span class="title">IPermissionChecker</span></span><br><span class="line">{</span><br><span class="line"> <span class="function"><span class="keyword">public</span> Task<<span class="built_in">bool</span>> <span class="title">IsGrantedAsync</span>(<span class="params"><span class="built_in">string</span> name</span>)</span></span><br><span class="line"> {</span><br><span class="line"> <span class="keyword">return</span> TaskCache.TrueResult;</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">public</span> Task<<span class="built_in">bool</span>> <span class="title">IsGrantedAsync</span>(<span class="params">ClaimsPrincipal? claimsPrincipal, <span class="built_in">string</span> name</span>)</span></span><br><span class="line"> {</span><br><span class="line"> <span class="keyword">return</span> TaskCache.TrueResult;</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">public</span> Task<MultiplePermissionGrantResult> <span class="title">IsGrantedAsync</span>(<span class="params"><span class="built_in">string</span>[] names</span>)</span></span><br><span class="line"> {</span><br><span class="line"> <span class="keyword">return</span> IsGrantedAsync(<span class="literal">null</span>, names);</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">public</span> Task<MultiplePermissionGrantResult> <span class="title">IsGrantedAsync</span>(<span class="params">ClaimsPrincipal? claimsPrincipal, <span class="built_in">string</span>[] names</span>)</span></span><br><span class="line"> {</span><br><span class="line"> <span class="keyword">return</span> Task.FromResult(<span class="keyword">new</span> MultiplePermissionGrantResult(names, PermissionGrantResult.Granted));</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>再到<code>ApplicationModule.cs</code>中設定要以<code>MyPermissionChecker</code>取代原生的<code>PermissionChecker</code>，<br><code>CustomAuthorAppService</code>取代原本的<code>AuthorAppService</code>，如下，</p><figure class="highlight csharp"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">public</span> <span class="keyword">override</span> <span class="keyword">void</span> <span class="title">ConfigureServices</span>(<span class="params">ServiceConfigurationContext context</span>)</span></span><br><span class="line">{</span><br><span class="line"> <span class="comment">//...其他程式 ...</span></span><br><span class="line"> context.Services.Replace(ServiceDescriptor.Singleton<IPermissionChecker, MyPermissionChecker>());</span><br><span class="line"> context.Services.Replace(ServiceDescriptor.Transient<IAuthorAppService, CustomAuthorAppService>());</span><br><span class="line">}</span><br></pre></td></tr></table></figure><h3 id="測試結果"><a href="#測試結果" class="headerlink" title="測試結果"></a>測試結果</h3><p>透過 Swagger 來測試時，在<code>MyPermissionChecker</code>的<code>IsGrantedAsync</code>方式設中斷點，<br>可以發現檢查順序為<br><code>BookStore.Authors</code>-><code>BookStore.Authors.Edit</code>-><code>BookStore.Authors.Edit2</code>-><code>BookStore.Authors.Edit</code>-><code>BookStore.Authors</code></p><p>所以可以發現，Method 上的<code>Authorize</code>的設定是疉加的效果。<br>所以如果是要覆寫的話，就使用原本的<code>Authorize</code>，不用特別再加一個新的<code>Authorize</code>。<br>或是新增另一個 Method 去設定不同的<code>Authorize</code>。<br>這個是 ASP.NET 原生的機制，不管是 Abp or ASP.NET 都是一樣的狀況哦。</p><h3 id="參考資料"><a href="#參考資料" class="headerlink" title="參考資料"></a>參考資料</h3><p><a href="https://github.com/abpframework/abp/blob/8.3.0/framework/src/Volo.Abp.Authorization/Volo/Abp/Authorization/Permissions/PermissionChecker.cs">abp PermissionChecker.cs</a><br><a href="https://github.com/abpframework/abp/blob/8.3.0/framework/src/Volo.Abp.Authorization.Abstractions/Volo/Abp/Authorization/Permissions/AlwaysAllowPermissionChecker.cs">abp AlwaysAllowPermissionChecker.cs</a></p>]]></content>
<summary type="html"><h3 id="問題"><a href="#問題" class="headerlink" title="問題"></a>問題</h3><p>同事繼承 Abp 範例中的 AuthorAppService ，將 UpdateAsync 設定成 virtual ，允許子類別去覆寫。<b</summary>
<category term="ASP.NET" scheme="https://rainmakerho.github.io/tags/ASP-NET/"/>
<category term="Authorize" scheme="https://rainmakerho.github.io/tags/Authorize/"/>
<category term="abp" scheme="https://rainmakerho.github.io/tags/abp/"/>
<category term="AppService" scheme="https://rainmakerho.github.io/tags/AppService/"/>
<category term="Policy" scheme="https://rainmakerho.github.io/tags/Policy/"/>
<category term="IPermissionChecker" scheme="https://rainmakerho.github.io/tags/IPermissionChecker/"/>
</entry>
<entry>
<title>ASP.NET [AllowAnonymous] 的覆寫問題</title>
<link href="https://rainmakerho.github.io/2024/10/30/aspnet-allowanonymous-class-level/"/>
<id>https://rainmakerho.github.io/2024/10/30/aspnet-allowanonymous-class-level/</id>
<published>2024-10-30T03:12:33.000Z</published>
<updated>2024-10-30T03:43:29.480Z</updated>
<content type="html"><![CDATA[<h3 id="使用-AllowAnonymous-與-Authorize-在-Class-Level-差異"><a href="#使用-AllowAnonymous-與-Authorize-在-Class-Level-差異" class="headerlink" title="使用 AllowAnonymous 與 Authorize 在 Class Level 差異"></a>使用 AllowAnonymous 與 Authorize 在 Class Level 差異</h3><p>在 Class 上設定 <code>AllowAnonymous</code> 屬性，跟設定 <code>Authorize</code> 效果相同嗎?<br>以下說明</p><h4 id="AllowAnonymous-在-Class-Level"><a href="#AllowAnonymous-在-Class-Level" class="headerlink" title="[AllowAnonymous] 在 Class Level"></a><code>[AllowAnonymous]</code> 在 Class Level</h4><figure class="highlight csharp"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line">[<span class="meta">AllowAnonymous</span>]</span><br><span class="line"><span class="keyword">public</span> <span class="keyword">class</span> <span class="title">PublicController</span> : <span class="title">Controller</span></span><br><span class="line">{</span><br><span class="line"> <span class="function"><span class="keyword">public</span> IActionResult <span class="title">Index</span>()</span></span><br><span class="line"> {</span><br><span class="line"> <span class="keyword">return</span> View();</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> [<span class="meta">Authorize</span>]</span><br><span class="line"> <span class="function"><span class="keyword">public</span> IActionResult <span class="title">Secure</span>()</span></span><br><span class="line"> {</span><br><span class="line"> <span class="keyword">return</span> View();</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>雖然 Method Secure 設定 <code>[Authorize]</code>，但它會被 Class 的 <code>[AllowAnonymous]</code>，<br>所以 Secure Method 還是允許匿名訪問。</p><h4 id="Authorize-在-Class-Level"><a href="#Authorize-在-Class-Level" class="headerlink" title="[Authorize] 在 Class Level"></a><code>[Authorize]</code> 在 Class Level</h4><figure class="highlight csharp"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line">[<span class="meta">Authorize</span>]</span><br><span class="line"><span class="keyword">public</span> <span class="keyword">class</span> <span class="title">SecureController</span> : <span class="title">Controller</span></span><br><span class="line">{</span><br><span class="line"> <span class="function"><span class="keyword">public</span> IActionResult <span class="title">Index</span>()</span></span><br><span class="line"> {</span><br><span class="line"> <span class="keyword">return</span> View();</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> [<span class="meta">AllowAnonymous</span>]</span><br><span class="line"> <span class="function"><span class="keyword">public</span> IActionResult <span class="title">Public</span>()</span></span><br><span class="line"> {</span><br><span class="line"> <span class="keyword">return</span> View();</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>整個 Class 設定為 <code>[Authorize]</code>，除了 <code>Public</code> Method 設定 <code>[AllowAnonymous]</code> 會覆寫 <code>[Authorize]</code>，所以允許匿名訪問。</p><p>詳細請參考 <a href="https://learn.microsoft.com/zh-tw/aspnet/core/diagnostics/asp0026">ASP0026：[AllowAnonymous] 從「更遠」覆寫 [Authorize]</a></p><p>所以在使用 <code>[AllowAnonymous]</code> 時，要小心哦~</p><h3 id="參考資源"><a href="#參考資源" class="headerlink" title="參考資源"></a>參考資源</h3><p><a href="https://learn.microsoft.com/zh-tw/aspnet/core/diagnostics/asp0026">ASP0026：[AllowAnonymous] 從「更遠」覆寫 [Authorize]</a></p>]]></content>
<summary type="html"><h3 id="使用-AllowAnonymous-與-Authorize-在-Class-Level-差異"><a href="#使用-AllowAnonymous-與-Authorize-在-Class-Level-差異" class="headerlink" title="</summary>
<category term="AllowAnonymous" scheme="https://rainmakerho.github.io/tags/AllowAnonymous/"/>
<category term="Authorize" scheme="https://rainmakerho.github.io/tags/Authorize/"/>
<category term="ASP0026" scheme="https://rainmakerho.github.io/tags/ASP0026/"/>
</entry>
<entry>
<title>Checkmarx | Permissive Content Security Policy</title>
<link href="https://rainmakerho.github.io/2024/10/07/checkmarx-permissive-content-security-policy/"/>
<id>https://rainmakerho.github.io/2024/10/07/checkmarx-permissive-content-security-policy/</id>
<published>2024-10-07T06:29:58.000Z</published>
<updated>2024-10-07T07:57:36.218Z</updated>
<content type="html"><![CDATA[<h3 id="Issue"><a href="#Issue" class="headerlink" title="Issue"></a>Issue</h3><p>ASP.NET Core Web 專案中，加入許多 Header 時，其中包含<strong>Content-Security-Policy</strong>，<br>被 Checkmarx 掃出有<strong>Permissive Content Security Policy</strong>的問題，程式碼如下，</p><figure class="highlight csharp"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">context.Response.Headers.Add(<span class="string">"X-Content-Security-Policy"</span>, <span class="string">"default-src 'self' http://localhost:61161; img-src 'self' blob: data:; frame-ancestors 'self'; form-action 'self'; object-src 'none';"</span>);</span><br><span class="line">context.Response.Headers.Add(<span class="string">"X-Frame-Options"</span>, <span class="string">"DENY"</span>);</span><br><span class="line">context.Response.Headers.Add(<span class="string">"Strict-Transport-Security"</span>, <span class="string">"max-age=31536000; includeSubDomains; preload"</span>);</span><br><span class="line">context.Response.Headers.Add(<span class="string">"X-Content-Type-Options"</span>, <span class="string">"nosniff"</span>);</span><br></pre></td></tr></table></figure><p>但 Checkmarx 指的位置卻是在<br><code>context.Response.Headers.Add("X-Content-Type-Options", "nosniff");</code><br><code>context.Response.Headers.Add("Strict-Transport-Security", "max-age=31536000; includeSubDomains; preload");</code><br>…其他非<code>X-Content-Security-Policy</code> Header 的地方。</p><h3 id="解法"><a href="#解法" class="headerlink" title="解法"></a>解法</h3><p>查看 Checkmarx 的 Rule 會檢查是否有 <strong>Content-Security-Policy</strong> Header, 值是否有 <code>default-src *</code> ，<code>form-action *</code>，<code>frame-ancestors *</code> ，才會有問題。</p><p>詢問同事 Jer 建議將 <strong>Content-Security-Policy</strong> 設定移到最後，以避免 Checkmarx 的誤判，</p><figure class="highlight csharp"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">context.Response.Headers.Add(<span class="string">"X-Frame-Options"</span>, <span class="string">"DENY"</span>);</span><br><span class="line">context.Response.Headers.Add(<span class="string">"Strict-Transport-Security"</span>, <span class="string">"max-age=31536000; includeSubDomains; preload"</span>);</span><br><span class="line">context.Response.Headers.Add(<span class="string">"X-Content-Type-Options"</span>, <span class="string">"nosniff"</span>);</span><br><span class="line">context.Response.Headers.Add(<span class="string">"X-Content-Security-Policy"</span>, <span class="string">"default-src 'self' http://localhost:61161; img-src 'self' blob: data:; frame-ancestors 'self'; form-action 'self'; object-src 'none';"</span>);</span><br></pre></td></tr></table></figure><p>再掃一次就沒有<strong>Permissive Content Security Policy</strong>的問題了。<br>或是將<code>context.Response.Headers.Add</code>改成<code>context.Response.Headers.Append</code>也 OK 哦~</p><h3 id="環境"><a href="#環境" class="headerlink" title="環境"></a>環境</h3><ul><li>Checkmarx 版本：9.5.5.1007 HF14</li><li>類別：Permissive Content Security Policy</li><li>嚴重程度：低風險</li></ul>]]></content>
<summary type="html"><h3 id="Issue"><a href="#Issue" class="headerlink" title="Issue"></a>Issue</h3><p>ASP.NET Core Web 專案中，加入許多 Header 時，其中包含<strong>Content-Sec</summary>
<category term="Content-Security-Policy" scheme="https://rainmakerho.github.io/tags/Content-Security-Policy/"/>
<category term="Checkmarx" scheme="https://rainmakerho.github.io/tags/Checkmarx/"/>
<category term="Permissive Content Security Policy" scheme="https://rainmakerho.github.io/tags/Permissive-Content-Security-Policy/"/>
</entry>
</feed>