repacking the modified squashfs into a deployable firmware file

[zengelan]

Hi,
I tried to understand the unpack script to create a "repack" script. But i am struggling with the crc calculations. Would it be generally feasible to write a scipt / tool which would re-pack a modified squashfas image to a deploymabel firmware file?
The goal is to modify the image file and upload it to the camrea without the need to have a serial connection.

regards
.andreas

[hn]

By coincidence, I reverse engineered the CRC check (a slightly modified CRC32 with poly 0x77073096) three days ago, see 9b966e2 :-)

Please contribute back things you create, a repack script might be helpful for others.

[hn]

To further add to your question: having a correct image file might not be sufficient. You have to understand the update procedure. The web gui internally calls the updater like this:

/mnt/app/update <number> imgfile [loader] [ext] [uitron] [uboot] [bootargs] [kernel] [fs] [app] [para] [NOUSE] [ptzmcu] [all]

Where <number> might be a version number and [loader] [ext] .. [all] might be a limitation on which parts are to be flashed. Maybe the updater compares versions numbers to decide which parts are to be flashed.

Inside the dvr binary you also find:

/mnt/tmp/update %d %s exceptboot
/mnt/tmp/update %d %s all
/mnt/app/update 0 %s app

Firmware v3.0.0.20_20052300 adds an /mnt/app/upgrade binary (what's the difference between update an upgrade?).

[zengelan]

we could be lucky due to the simple image layout. i recorded a log of doing an update/upgrade (they use both terms). The image layout is the following as printed out by the update task:

---image layer out---
loader in /dev/mtd9     0x00000000      +0x0    0x10000
ext in /dev/mtd9        0x00010000      +0x10000        0x10000
uitron in /dev/mtd9     0x00020000      +0x20000        0x3a0000
uboot in /dev/mtd9      0x003c0000      +0x3c0000       0x50000
bootargs in /dev/mtd9   0x00410000      +0x410000       0x10000
kernel in /dev/mtd9     0x00420000      +0x420000       0x200000
fs in /dev/mtd9 0x00620000      +0x620000       0x9sh (823): drop_caches: 1
00000
app in /dev/mtd9        0xffffffff      +0xffffffff     0x0
para in /dev/mtd9       0x00f20000      +0xf20000       0xc0000
NOUSE in /dev/mtd9      0xffffffff      +0xffffffff     0x0
ptzmcu in /dev/mtd9     0xffffffff      +0xffffffff     0x0
---------------------

img section:
section:loader start:0 len:10000
section:ext start:10000 len:10000
section:uitron start:20000 len:3a0000
section:uboot start:3c0000 len:50000
section:bootargs start:410000 len:10000
section:kernel start:420000 len:200000
section:fs start:620000 len:900000
section:app start:ffffffff len:0
section:para start:f20000 len:c0000
section:NOUSE start:ffffffff len:0
section:ptzmcu start:ffffffff len:0

reolink update log.txt

the squashfs image is at section:fs start:620000 with length:900000

so it might be as simple as adding the new squafsh image at the place 0x620000 / the place where the previous is, even if its bigger / smaller as always the full range will be written to NOR.

see ref prepare update fs to mtd:/dev/mtd9 in the attached file. in tghe log is shows that is blindly erases and writes 0x72A000 bytes from offset 0x620000

(Note with my camera, i found the squafsh image offset at 0x620000 which was different to what you had hardcoded in as 0x6e0000. seems these offsets are either different by camera, or it has chnaged in the recent fw version that I applied a few days ago before starting reverse engineering

i found the right offset from the uboot commandline:

cmdline = earlyprintk console=ttyS0,115200 rootwait rootfstype=squashfs ro  mtdparts=spi_nor.0:0x10000@0x0(loader),0x10000@0x10000(modelext),0x3a0000@0x20000(u               itron),0x50000@0x3c0000(uboot),0x10000@0x410000(uenv),0x200000@0x420000(linux),0x900000@0x620000(rootfs),0xc0000@0xf20000(para),0x20000@0xfe0000(sp),0x1000000@0x0(all)  root=/dev/mtdblock6  mem=0x7ff000@0x101000 mem=0x3b00000@0x900000 sdio_driving=20,15,15,20,15,15,30,25,25,30,25,25 ethaddr=xxx

see 0x900000@0x620000(rootfs)
)

[zengelan]

some more research on the theory above (the squashfas is simply the last binary blob at the end of the image file and can be replaced)

doing a binwalk on an orignal firmware file

• https://support.reolink.com/hc/en-us/articles/900001840706-07-10-2020-Firmware-for-Reolink-IP-Cameras-IPC-51516M-
• RLC-410-5MP
• https://reolink-storage.s3.amazonaws.com/website/firmware/20200721firmware/RLC-410-5MP_65_20071000.zip
  

reveals:

> binwalk IPC_51516M5M.65_20071000.RLC-410-5MP.OV05A10.5MP.REOLINK.pak

DECIMAL         HEX             DESCRIPTION
-------------------------------------------------------------------------------------------------------
35024           0x88D0          LZMA compressed data, properties: 0x03, dictionary size: 33554432 bytes, uncompressed size: 973078528 bytes
41976           0xA3F8          LZMA compressed data, properties: 0x03, dictionary size: 33554432 bytes, uncompressed size: 973078528 bytes
2576679         0x275127        LZMA compressed data, properties: 0x04, dictionary size: 16777216 bytes, uncompressed size: 33554432 bytes
2681056         0x28E8E0        LZMA compressed data, properties: 0x02, dictionary size: 33554432 bytes, uncompressed size: 50331648 bytes
2681112         0x28E918        LZMA compressed data, properties: 0x02, dictionary size: 33554432 bytes, uncompressed size: 50331648 bytes
..........
3016108         0x2E05AC        LZMA compressed data, properties: 0x02, dictionary size: 33554432 bytes, uncompressed size: 16777216 bytes
3016556         0x2E076C        LZMA compressed data, properties: 0x04, dictionary size: 33554432 bytes, uncompressed size: 16777216 bytes
3018744         0x2E0FF8        LZMA compressed data, properties: 0x01, dictionary size: 16777216 bytes, uncompressed size: 50331648 bytes
3018788         0x2E1024        LZMA compressed data, properties: 0x01, dictionary size: 33554432 bytes, uncompressed size: 50331648 bytes
3026796         0x2E2F6C        LZMA compressed data, properties: 0x01, dictionary size: 33554432 bytes, uncompressed size: 268435456 bytes
3026824         0x2E2F88        LZMA compressed data, properties: 0x01, dictionary size: 16777216 bytes, uncompressed size: 33554432 bytes
3030345         0x2E3D49        LZMA compressed data, properties: 0x04, dictionary size: 8388608 bytes, uncompressed size: 65011712 bytes
3078388         0x2EF8F4        LZMA compressed data, properties: 0x02, dictionary size: 16777216 bytes, uncompressed size: 50331648 bytes
3083529         0x2F0D09        LZMA compressed data, properties: 0x31, dictionary size: 65536 bytes, uncompressed size: 131072 bytes
..........
3112813         0x2F7F6D        LZMA compressed data, properties: 0x30, dictionary size: 65536 bytes, uncompressed size: 3407872 bytes
3114016         0x2F8420        LZMA compressed data, properties: 0x01, dictionary size: 16777216 bytes, uncompressed size: 251658240 bytes
3114052         0x2F8444        LZMA compressed data, properties: 0x01, dictionary size: 16777216 bytes, uncompressed size: 251658240 bytes
3114088         0x2F8468        LZMA compressed data, properties: 0x01, dictionary size: 16777216 bytes, uncompressed size: 251658240 bytes
3114873         0x2F8779        LZMA compressed data, properties: 0x04, dictionary size: 65536 bytes, uncompressed size: 589824000 bytes
3188688         0x30A7D0        LZMA compressed data, properties: 0x02, dictionary size: 16777216 bytes, uncompressed size: 33554432 bytes
3188808         0x30A848        LZMA compressed data, properties: 0x02, dictionary size: 16777216 bytes, uncompressed size: 33554432 bytes
3189252         0x30AA04        LZMA compressed data, properties: 0x13, dictionary size: 16777216 bytes, uncompressed size: 100663296 bytes
3189332         0x30AA54        LZMA compressed data, properties: 0x13, dictionary size: 16777216 bytes, uncompressed size: 100663296 bytes
3192656         0x30B750        LZMA compressed data, properties: 0x02, dictionary size: 16777216 bytes, uncompressed size: 268435456 bytes
3192901         0x30B845        LZMA compressed data, properties: 0x01, dictionary size: 524288 bytes, uncompressed size: 8388608 bytes
3239197         0x316D1D        LZMA compressed data, properties: 0x03, dictionary size: 4194304 bytes, uncompressed size: 7733248 bytes
3243132         0x317C7C        LZMA compressed data, properties: 0x0A, dictionary size: 16777216 bytes, uncompressed size: 67108864 bytes
3245676         0x31866C        LZMA compressed data, properties: 0x04, dictionary size: 33554432 bytes, uncompressed size: 16777216 bytes
3245700         0x318684        LZMA compressed data, properties: 0x04, dictionary size: 33554432 bytes, uncompressed size: 16777216 bytes
..........
3255952         0x31AE90        LZMA compressed data, properties: 0x01, dictionary size: 33554432 bytes, uncompressed size: 16777216 bytes
3256276         0x31AFD4        LZMA compressed data, properties: 0x01, dictionary size: 33554432 bytes, uncompressed size: 16777216 bytes
3328653         0x32CA8D        LZMA compressed data, properties: 0x88, dictionary size: 262144 bytes, uncompressed size: 5669 bytes
3476844         0x350D6C        lzop compressed data - version 0.000,
3529392         0x35DAB0        uImage header, header size: 64 bytes, header CRC: 0x637D88B0, created: Tue Feb 26 11:19:51 2019, image size: 1634561 bytes, Data Address: 0x80101000, Entry Point: 0x804105D0, data CRC: 0x8470FA3B, OS: Linux, CPU: MIPS, image type: OS Kernel Image, compression type: lzma, image name: "Linux-4.1.0"
5164017         0x4ECBF1        Squashfs filesystem, little endian, version 4.0, compression:  size: 7510470 bytes,  653 inodes, blocksize: 65536 bytes, created: Fri Jul 10 04:54:42 2020

see the last row where the plain squashfs image was found at offset 0x4ECBF1. Nothing else reported after that

5164017         5164017         Squashfs filesystem, little endian, version 4.0, compression:  size: 7510470 bytes,  653 inodes, blocksize: 65536 bytes, created: Fri Jul 10 04:54:42 2020

hexdumping this with offset 5164017:

> hexdump -s 5164017 IPC_51516M5M.65_20071000.RLC-410-5MP.OV05A10.5MP.REOLINK.pak >hexdump_squashfs_source.txt
> ls
-rw-r--r--  1 root root 22531498 Jul 26 22:42 hexdump_squashfs_source.txt
-rw-r--r--  1 root root 12676081 Jul 18 09:15 IPC_51516M5M.65_20071000.RLC-410-5MP.OV05A10.5MP.REOLINK.pak

now extracting the images using the unpack-novatek-firmware.pl (link to file) tool from this repo:

>./unpack-novatek-firmware.pl -w IPC_51516M5M.65_20071000.RLC-410-5MP.OV05A10.5MP.REOLINK.pak
 Finished reading header
 Finished reading payload
 Closed file
 Binheader matches magic
 loaded btype
 loaded btypenulled
Board/File-Type: 02240000
Board/File-Type with 0x00@pos2: 02000000
File CRC from binheader: 8308ef0b
Calc CRC: 8308ef0b
File CRC and computed CRC match, everything is ok.
Image File Section  0    name: loader
Image File Section  0 version: v1.0.0.1
Image File Section  0  offset:     1552
Image File Section  0  length:    32768
Writing output file 'IPC_51516M5M.65_20071000.RLC-410-5MP.OV05A10.5MP.REOLINK-partition-0-loader.bin'

Image File Section  1    name: ext
Image File Section  1 version: v1.0.0.1
Image File Section  1  offset:    34320
Image File Section  1  length:     2856
Writing output file 'IPC_51516M5M.65_20071000.RLC-410-5MP.OV05A10.5MP.REOLINK-partition-1-ext.bin'

Image File Section  2    name: uitron
Image File Section  2 version: v1.0.0.1
Image File Section  2  offset:    37176
Image File Section  2  length:  3229552
Writing output file 'IPC_51516M5M.65_20071000.RLC-410-5MP.OV05A10.5MP.REOLINK-partition-2-uitron.bin'

Image File Section  3    name: uboot
Image File Section  3 version: v1.0.0.1
Image File Section  3  offset:  3266728
Image File Section  3  length:   262664
Writing output file 'IPC_51516M5M.65_20071000.RLC-410-5MP.OV05A10.5MP.REOLINK-partition-3-uboot.bin'

Image File Section  4    name:
Image File Section  4 version:
Image File Section  4  offset:  3529392
Image File Section  4  length:        0

Image File Section  5    name: kernel
Image File Section  5 version: v1.0.0.1
Image File Section  5  offset:  3529392
Image File Section  5  length:  1634625
Writing output file 'IPC_51516M5M.65_20071000.RLC-410-5MP.OV05A10.5MP.REOLINK-partition-5-kernel.bin'

Image File Section  6    name: fs
Image File Section  6 version: v1.0.0.1
Image File Section  6  offset:  5164017
Image File Section  6  length:  7512064
Writing output file 'IPC_51516M5M.65_20071000.RLC-410-5MP.OV05A10.5MP.REOLINK-partition-6-fs.bin'

Image File Section  7    name:
Image File Section  7 version:
Image File Section  7  offset: 12676081
Image File Section  7  length:        0

Image File Section  8    name:
Image File Section  8 version:
Image File Section  8  offset: 12676081
Image File Section  8  length:        0

Image File Section  9    name:
Image File Section  9 version:
Image File Section  9  offset: 12676081
Image File Section  9  length:        0

Image File Section 10    name:
Image File Section 10 version:
Image File Section 10  offset: 12676081
Image File Section 10  length:        0

Partition  0    name: loader
Partition  0     dst: /dev/mtd9
Partition  0  offset:         0
Partition  0 unknown:         0
Partition  0    size:     65536

Partition  1    name: ext
Partition  1     dst: /dev/mtd9
Partition  1  offset:     65536
Partition  1 unknown:     65536
Partition  1    size:     65536

Partition  2    name: uitron
Partition  2     dst: /dev/mtd9
Partition  2  offset:    131072
Partition  2 unknown:    131072
Partition  2    size:   3801088

Partition  3    name: uboot
Partition  3     dst: /dev/mtd9
Partition  3  offset:   3932160
Partition  3 unknown:   3932160
Partition  3    size:    327680

Partition  4    name: bootargs
Partition  4     dst: /dev/mtd9
Partition  4  offset:   4259840
Partition  4 unknown:   4259840
Partition  4    size:     65536

Partition  5    name: kernel
Partition  5     dst: /dev/mtd9
Partition  5  offset:   4325376
Partition  5 unknown:   4325376
Partition  5    size:   2097152

Partition  6    name: fs
Partition  6     dst: /dev/mtd9
Partition  6  offset:   6422528
Partition  6 unknown:   6422528
Partition  6    size:   9437184

Partition  7    name: app
Partition  7     dst: /dev/mtd9
Partition  7  offset:        -1
Partition  7 unknown:        -1
Partition  7    size:         0

Partition  8    name: para
Partition  8     dst: /dev/mtd9
Partition  8  offset:  15859712
Partition  8 unknown:  15859712
Partition  8    size:    786432

Partition  9    name: NOUSE
Partition  9     dst: /dev/mtd9
Partition  9  offset:        -1
Partition  9 unknown:        -1
Partition  9    size:         0

Partition 10    name: ptzmcu
Partition 10     dst: /dev/mtd9
Partition 10  offset:        -1
Partition 10 unknown:        -1
Partition 10    size:         0

The squashfs image file is extracted correctly at offset 5164017 as IPC_51516M5M.65_20071000.RLC-410-5MP.OV05A10.5MP.REOLINK-partition-6-fs.bin:

> ll
-rw-r--r--  1 root root 12676081 Jul 18 09:15 IPC_51516M5M.65_20071000.RLC-410-5MP.OV05A10.5MP.REOLINK.pak
-rw-r--r--  1 root root  7512064 Jul 26 22:31 IPC_51516M5M.65_20071000.RLC-410-5MP.OV05A10.5MP.REOLINK-partition-6-fs.bin

now hexdumping this extracted file,too:

> hexdump IPC_51516M5M.65_20071000.RLC-410-5MP.OV05A10.5MP.REOLINK-partition-6-fs.bin >IPC_51516M5M.65_20071000.RLC-410-5MP.OV05A10.5MP.REOLINK-partition-6-fs.txt
-rw-r--r--  1 root root 22531498 Jul 26 22:44 IPC_51516M5M.65_20071000.RLC-410-5MP.OV05A10.5MP.REOLINK-partition-6-fs.txt

file size is the same, and looks like the same output:

>ll *.txt
-rw-r--r-- 1 root root 22531498 Jul 26 22:42 hexdump_squashfs_source.txt
-rw-r--r-- 1 root root 22531498 Jul 26 22:44 IPC_51516M5M.65_20071000.RLC-410-5MP.OV05A10.5MP.REOLINK-partition-6-fs.txt

the addresses are different, so can't do a diff.
first 5 and last 5 lines of hexdump_squashfs_source.txt:

04ecbf1 7368 7371 028d 0000 d872 5f07 0000 0001
04ecc01 003b 0000 0004 0010 00c0 0001 0004 0000
04ecc11 1b3b 10b0 0000 0000 99c6 0072 0000 0000
04ecc21 99be 0072 0000 0000 ffff ffff ffff ffff
04ecc31 6502 0072 0000 0000 7a94 0072 0000 0000
.....
0c165a1 95ee 0072 0000 0000 8004 0400 0000 99b8
0c165b1 0072 0000 0000 0000 0000 0000 0000 0000
0c165c1 0000 0000 0000 0000 0000 0000 0000 0000
*
0c16bf1

first 5 and last 5 lines of IPC_51516M5M.65_20071000.RLC-410-5MP.OV05A10.5MP.REOLINK-partition-6-fs.txt:

0000000 7368 7371 028d 0000 d872 5f07 0000 0001
0000010 003b 0000 0004 0010 00c0 0001 0004 0000
0000020 1b3b 10b0 0000 0000 99c6 0072 0000 0000
0000030 99be 0072 0000 0000 ffff ffff ffff ffff
0000040 6502 0072 0000 0000 7a94 0072 0000 0000
.....
07299b0 95ee 0072 0000 0000 8004 0400 0000 99b8
07299c0 0072 0000 0000 0000 0000 0000 0000 0000
07299d0 0000 0000 0000 0000 0000 0000 0000 0000
*
072a000

this also looks similar. so now doing a binary compare.
First unpack the raw binary data from the image file:

>  dd skip=5164017 bs=1 if=IPC_51516M5M.65_20071000.RLC-410-5MP.OV05A10.5MP.REOLINK.pak of=last_chunk.bin
7512064+0 records in
7512064+0 records out
7512064 bytes (7,5 MB) copied, 6,70605 s, 1,1 MB/s
> ll last_chunk.bin
-rw-r--r--  1 root root  7512064 Jul 26 22:55 last_chunk.bin

performing a diff:

> diff -s last_chunk.bin IPC_51516M5M.65_20071000.RLC-410-5MP.OV05A10.5MP.REOLINK-partition-6-fs.bin
Files last_chunk.bin and IPC_51516M5M.65_20071000.RLC-410-5MP.OV05A10.5MP.REOLINK-partition-6-fs.bin are identical

this proves that the squashfs image is indeed the last chunk in the firmware update file. It should be possible to overwrite the last chunks starting from the offset where the squashfs header is found (in this case 5164017 / 0x4ECBF1) with a new squashfs image. And even if the new image is longer.

The tricky part that is still outstanding, is to calculate the checksum of a new squashfs image and modify the header of the file accordingly.

[hn]

fs partition is actually given wrongly in the output as 6422528

No, the fs partiton is correctly displayed:

Image File Section 6 offset: 5164017

You refer 6422528, which is the offset on flash:

Partition 6 offset: 6422528

The pak file consists of both the image file section and (flash) partition layouts.

[zengelan]

No, the fs partiton is correctly displayed:

you are correct, the partition offset is correctly displayed. Comment editd

[hn]

Thanks for the hint regarding the fs flash position. It seems that they changed the position starting with firmware 65_20071000, see *.pak files:

Firmware 20_20052300 (and earlier): Partition 6 offset: 7208960 (= 0x6e0000)
Firmware 65_20071000: Partition 6 offset: 6422528 (= 0x620000)

[zengelan]

Wanted to give an update on this. I made significant progress in re-packing the modified squashfs into a pak file. I'm planning to upload a pull request for unpack-novatek-firmware.pl in the next days. This includes a flag that will take the original pak file and a mtdblock6-NEW.bin file and will write a new pak fiele with proper headers and CRC that can be deployed.
I deployed this on two of my cameras, which had older versions, without issues and port 22 is now open and listening.... but for some reason I cannot connect via ssh. Doing this results in the following after about 30 seconds

ssh -oHostKeyAlgorithms=ssh-rsa root@192.168.7.84
Connection closed by 192.168.7.84

I now have a bit of trouble as I haven't found a way to upgrade the firmware with the same version again, the UI keeps complaining. Also i can't do a downgrade, Maybe I should have made a dump on sdcard before flashing...

[hn]

Try to start dropbear in foreground mode (-F -E) and see if any error is reported.

As I said in #4 (comment) we have to learn more about the update process (version numbers, ...).

[zengelan]

but for some reason I cannot connect via ssh

I used the wrong dropbox binary, had to use the one from the original contrib directory with a different path for the hostkey location.
created pull request for injecting the modified squashfs to an pak image file. #5

[t-jones14]

Any update on this?

[hn]

I'm not merging zengelan's repack-script into the main tree because the update mechanism in all its entirety hasn't been understood well (at least by me). For those who are brave of heart there is no known reason for not using the script.

[GhostlyCrowd]

@zengelan I've tried your script because i can always dump and reflash the soc with my tools if it goes sideway.

I cannot get it to update through the web UI though. it kicks back an "unknown error during update, please try again after device restarts"

[GhostlyCrowd]

File CRC: ef16b88b Calc CRC: a440fb10 Warning: File CRC does NOT match computed CRC value!

That's what the repack tells me when I try to unpack it again which I'm sure is why the flash fails.

Edit: I had to fix the CRC bytes in the header of the repacked file with a hex editor and after naming it appropriately to pass their lame version checking it has flashed and is working.

Latest Firmware for the RLC-410-5MP v3.0.0.136_20121100