Large file save issue

[sheckandar]

I'm trying to add a large file (70 GiB) to a cookfs archive and encountering an issue where tclsh runs out of memory and then is killed by the system.

The file in question is being saved in 4K chunks. The problem is that cookfs never flushes data to disk causing all the data to buffer in RAM. Flush seems to happen only when close $image command is issued.

I have a RedHat 8 system with TCL 9.0.3 and running cookfs from the master branch. The following script showcases the problem.

set cfsid [cookfs::Mount -volume /media/images/test.cfs //test:/]
set image [open //test:/test.img wb]
set device [open /dev/rhel/root rb]
chan configure $device -blocking 0

while {![chan eof $device]} {
   set block [chan read $device 4096]
   #more code here
   chan puts -nonewline $image $block
   chan flush $image
}

The file /media/images/test.cfs stays at 0 bytes on disk and the memory usage keeps going up steadily, until all the RAM is used up or close command is issued. The flush command has no effect and neither does setting channel buffering to none.

[sheckandar]

After looking thought the code this seems to be by design (using memchan) and not a bug. The workaround is to use pages and index commands, I suppose. This is not ideal as I have to write code that keeps track of pages, files and sizes, I would rather have cookfs keep track of all those.

I explored saving files to disk and then adding to a cookfs archive or repeatedly closing and reopening a file in append mode, but doing that is very time consuming and at least doubles the time required to generate an archive.

Can you suggest any other way to save a large file in chunks without filling up all the available RAM ?

[sheckandar]

So I was working with index commands and ran across a bug. This bug seems to happen if the number of pages exceeds 10919. The page size doesn't seem to matter.

Here's the test code:

set cookfs [cookfs::pages -compression zstd:4 /media/images/test.cfs]
set cookfsIdx [cookfs::fsindex [$cookfs index]]

for {set page 0} {$page < 10921} {incr page} {
	lappend index $page 0 262144
}

$cookfsIdx set test.img [clock seconds] $index
$cookfs index [$cookfsIdx export]
$cookfs close
$cookfsIdx delete
unset index

And the error I got:

malloc.c:2415: sysmalloc: Assertion `(old_top == initial_top (av) && old_size == 0) || ((unsigned long) (old_size) >= MINSIZE && prev_inuse (old_top) && ((unsigned long) old_end & (pagesize - 1)) == 0)' failed.

Or
corrupted size vs. prev_size

Or

realloc(): invalid next size                                                                                                                                                                                   
Aborted 

[sheckandar]

UPDATE

After spending some time looking through the source code, I finally pinpointed the issue causing realloc error.

It is on line 307 in fsindexIO.c file.

When Cookfs_Int2Binary function is called it corrupts result TCL array object.
This only happens when a number of pages increases above the number I mentioned in my previous post.

But, as I'm not very good with C, a fix would be appreciated.