-
Notifications
You must be signed in to change notification settings - Fork 1
/
Copy pathbuffer_internals.html
117 lines (117 loc) · 6.46 KB
/
buffer_internals.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
<!-- HTML header for doxygen 1.8.10-->
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/xhtml;charset=UTF-8"/>
<meta http-equiv="X-UA-Compatible" content="IE=9"/>
<meta name="generator" content="Doxygen 1.9.4"/>
<title>librsync: Buffer internals</title>
<link href="tabs.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="jquery.js"></script>
<script type="text/javascript" src="dynsections.js"></script>
<link href="doxygen.css" rel="stylesheet" type="text/css" />
</head>
<body>
<div id="top"><!-- do not remove this div, it is closed by doxygen! -->
<!-- ad -->
<script async src="//pagead2.googlesyndication.com/pagead/js/adsbygoogle.js"></script>
<!-- librsync -->
<ins class="adsbygoogle"
style="display:block"
data-ad-client="ca-pub-3547096055927362"
data-ad-slot="8322976738"
data-ad-format="auto"></ins>
<script>
(adsbygoogle = window.adsbygoogle || []).push({});
</script>
<div id="titlearea">
<table cellspacing="0" cellpadding="0">
<tbody>
<tr style="height: 56px;">
<td id="projectalign" style="padding-left: 0.5em;">
<div id="projectname">librsync
 <span id="projectnumber">2.3.4</span>
</div>
</td>
</tr>
</tbody>
</table>
</div>
<!-- end header part -->
<!-- Generated by Doxygen 1.9.4 -->
<script type="text/javascript" src="menudata.js"></script>
<script type="text/javascript" src="menu.js"></script>
<script type="text/javascript">
/* @license magnet:?xt=urn:btih:d3d9a9a6595521f9666a5e94cc830dab83b65699&dn=expat.txt MIT */
$(function() {
initMenu('',false,false,'search.php','Search');
});
/* @license-end */
</script>
<div id="main-nav"></div>
</div><!-- top -->
<div><div class="header">
<div class="headertitle"><div class="title">Buffer internals </div></div>
</div><!--header-->
<div class="contents">
<div class="textblock"><p ><a class="anchor" id="md_doc_buffer_internals"></a> </p>
<h1><a class="anchor" id="autotoc_md46"></a>
Input scoop</h1>
<p >A module called the <em>scoop</em> is used for buffering data going into librsync. It accumulates data when the application does not supply it in large enough chunks for librsync to make use of it.</p>
<p >The scoop object is a set of fields in the rs_job_t object:: </p><pre class="fragment">char *scoop_buf; /* the allocation pointer */
size_t scoop_alloc; /* the allocation size */
size_t scoop_avail; /* the data size */
</pre><p> Data from the read callback always goes into the scoop buffer.</p>
<p >The state functions call rs__scoop_read when they need some input data. If the read callback blocks, it might take multiple attempts before it can be filled. Each time, the state function will also need to block, and then be reawakened by the library.</p>
<p >Once the scoop has been sufficiently filled, it must be completely consumed by the state function. This is easy if the state function always requests one unit of work at a time: a block, a file header element, etc.</p>
<p >All this means that the valid data is always located at the start of the scoop, continuing for scoop_avail bytes. The library is never allowed to consume only part of the data.</p>
<p >One the state function has consumed the data, it should call rs__scoop_reset(), which resets scoop_avail to 0.</p>
<h1><a class="anchor" id="autotoc_md47"></a>
Output queue</h1>
<p >The library can set up data to be written out by putting a pointer/length for it in the output queue:: </p><pre class="fragment">char *outq_ptr;
size_t outq_bytes;
</pre><p> The job infrastructure will make sure this is written out before the next call into the state machine.</p>
<p >There is only one outq_ptr, so any given state function can only produce one contiguous block of output.</p>
<h1><a class="anchor" id="autotoc_md48"></a>
Buffer sharing</h1>
<p >The scoop buffer may be used by the output queue. This means that data can traverse the library with no extra copies: one copy into the scoop buffer, and one copy out. In this case outq_ptr points into scoop_buf, and outq_bytes tells how much data needs to be written.</p>
<p >The state function calls rs__scoop_reset before returning when it is finished with the data in the scoop. However, the outq may still point into the scoop buffer, if it has not yet been able to be copied out. This means that there is data in the scoop beyond scoop_avail that must still be retained.</p>
<p >This is safe because neither the scoop nor the state function will get to run before the output queue has completely drained.</p>
<h1><a class="anchor" id="autotoc_md49"></a>
Readahead</h1>
<p >How much readahead is required?</p>
<p >At the moment (??) our rollsum and MD4 routines require a full contiguous block to calculate a checksum. This could be relaxed, at a possible loss of efficiency.</p>
<p >So calculating block checksums requires one full block to be in memory.</p>
<p >When applying a patch, we only need enough readahead to unpack the command header.</p>
<p >When calculating a delta, we need a full block to calculate its checksum, plus space for the missed data. We can accumulate any amount of missed data before emitting it as a literal; the more we can accumulate the more compact the encoding will be. </p>
</div></div><!-- contents -->
</div><!-- PageDoc -->
<!-- HTML footer for doxygen 1.8.10-->
<!-- start footer part -->
<!-- ad -->
<script async src="//pagead2.googlesyndication.com/pagead/js/adsbygoogle.js"></script>
<!-- librsync -->
<ins class="adsbygoogle"
style="display:block"
data-ad-client="ca-pub-3547096055927362"
data-ad-slot="8322976738"
data-ad-format="auto"></ins>
<script>
(adsbygoogle = window.adsbygoogle || []).push({});
</script>
<!-- analytics -->
<script>
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
ga('create', 'UA-71109100-1', 'auto');
ga('send', 'pageview');
</script>
<hr class="footer"/><address class="footer"><small>
Generated on Sun Feb 19 2023 16:26:51 for librsync by  <a href="http://www.doxygen.org/index.html">
<img class="footer" src="doxygen.png" alt="doxygen"/>
</a> 1.9.4
</small></address>
</body>
</html>