Occaisional corruption of the downloaded hex file --> The hex file cannot be decoded. Checksum calculation failure occurred

[jaustin]

This is a tricky to track down issue, so hopefully posting it here we'll find other people with the same problem...

Though we haven't ever seen or reproduced it ourselves in the foundation, a number of people have sent us hex files that are missing large chunks of data, and the result is that the checksums fail and DAPLink refuses (correctly!) to flash the file.

The error DAPLink puts in the fail.txt is:
The hex file cannot be decoded. Checksum calculation failure occurred

Missing chunks are not on line boundaries.

[ntoll]

An example of several of these hex files would be very useful!

[jaustin]

Yes, we've got to get permission to share them, which is why I haven't yet uploaded one - as I said, we can't reproduce ourselves yet :/

[ntoll]

Ugh... I presume the permission is needed because this is the work of someone under 16?

[jaustin]

Well, in one case yes but irrespective of age, we don't share people's hex files in a public forum (and hence their code, could contain all sorts of things!) without first asking permission :)

…

On 12 Jul 2017 7:20 pm, "Nicholas Tollervey" ***@***.***> wrote: Ugh... I presume the permission is needed because this is the work of someone under 16? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#55 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAI-qV_gGrf2m2uBfeiVYRrO8ftBw7mlks5sNQ54gaJpZM4OWAKs> .

[jaustin]

We've got permission to share the following files - one is a corrupted hex and the other is the result of 'File-->Save as' on the page that was giving bad data.

The user believes there's no user visible virus software or browser security extensions and plugins.

confusing broken editor files.zip

[ZanderBrown]

Only contains one hex file, the other is just the .html of the editor.

Am I missing something or is the user attempting to download the hex via CTRL-S and saving the webpage rather than a hex generated by the webpage?

[ntoll]

So, I can confirm it's something at the user's end of things. I did a 3-way diff to tell this:

• Corrupt hex is corrupt as reported by the user. (A)
• Source of the editor containing the MicroPython runtime in a div contains exactly the same hex content apart from the user's script which is in the right place. (B) This means the editor did its job and inserted the user's script into the right location in the hex.
• The content of the two previous hex (A and B) are different to the hex in the published editor (C). I can confirm that C is working.

Put simply, the version of the editor on the user's local machine has a hex file embedded into the HTML that is corrupt.

The 3-way diff demonstrated that large chunks of the hex served as part of the editor (C) are missing and replaced by lines of the wrong length (i.e. they're not the expected uniform length before a newline).

While the user might not think there's anything interfering with their communications, the above proves conclusively that there is. The source code delivered by python.microbit.org is correct, but what they see is different, ergo, something is interfering.

Do they have a proxy? Have they manipulated the HTML in any way? Are they or their ISP using some sort of filter that manipulates the contents of HTTP responses? In any case, the problem is firmly in their realm. Perhaps clearing the caches and reloading might do the trick?

One way we could mitigate these problems is by checksumming the content of the hidden div and checking it in client-side JS when the user tries to make a hex file.

As for the user's next steps, depending on how technical they are, I'd advise them to check their network stack to ensure there's not a proxy, filter or some other thing interfering with things. This may be at the ISP level.

Other than that, there's nothing we can do if we're serving correct content. :-(

[jaustin]

Could we please leave this open and tagged 'needsmoreinfo' or something to suggest we don't believe it's explicitly a problem with the code itself? That way there's something visible for other users seeing the same problem to find it? I think we don't really understand it yet and if someone were to show up on GitHub with the same issue and some tools to help debug it might be really useful.

Your conclusion is the same as ours here, by the way - something somewhere is corrupting the HTML file and as a result the hex file.

One thing @microbit-matt and @carlosperate noticed was that Chrome 'chunks' the long string in the inspector view, maybe a possible target for investigation.

[carlosperate]

Did we end up getting the users' chrome version? This could be important to identify the problem, in case it's a browser bug.
Maybe this can be of some help as well: https://bugs.chromium.org/p/chromium/issues/detail?id=161127

[ntoll]

Reopened as per @jaustin's request.

[ntoll]

My biggest concern is that we're trying to fix a problem with a user's system over which we have no control.

The content is served correctly, via TCP/IP (so we can presume transit via the network isn't corrupting it). That leaves everything within the realm of the user. The best we can possibly do is simply flag that we can't do anything but point the user to check their own software/stack.

If it's a bug in Chrome, then it should be reported to Google in such a way that it's easy to recreate the problem condition. But is that the responsibility of the micro:bit foundation or the user who has the buggy browser?

Could they try it in another browser to confirm it's Chrome specific?

[carlosperate]

A few people have reported the same problem, and taking in consideration only a small % of people would actually make the effort to report it, imagine how many have just moved on and used something else. So perhaps we can mitigate it if we understand the problem a little bit better.

One thing we did wonder before is if the DOM was manipulated by any browser plug-in? Maybe a firewall or Ad-blocker. It's probably not the case, but if suddenly this didn't work for all uBlock users, then perhaps it's something we should address, so understanding why it's happening might still be worthwhile.

What you suggested earlier could be a way of mitigating it, perhaps fetching the HEX as an independent file and doing some type of checksum to ensure it is valid.

One disadvantage of doing this could be that we could lose the ability to allow users to just download a github zip and open the html file. If we definitely want to maintain this feature, we can still implement it in a way where we only do a XMLHttpRequest as a fallback (i.e., store the hex as a javascript string in a file, check the contents, and if needed then fetch again via XMLHttpRequest).

[fabgui]

Hello,

This post is fairly old but I might have a corrupted file for you.

I am very new to MicroBit but fairly computer competent (rapsbery pi, linux, python, ++). The kids brought two MicroBits back from school and played with them for a while. This evening, my daughter called after being unsuccesfull a downloading her latest hex file (enclosed). Windows was showing an 0x800703EE error and would not allow download of the file.
We tried:

• changing batteries
• changing battery pack
• downloading known-working HEX from the microbit website, called "first experience" and "smiley". Both failed to upload to the MicroBit
• downloading HEX file with the "Microsoft micro:bit uploader". No apparent error message but it did not bring the Microbit to life.
• upgrading to latest firmware. The firmware upgrade seem to have worked: new version in DETAILS.TXT but I was still unable to upload firmware.
• Clearing out the Microbit using: https://support.microbit.org/support/solutions/articles/19000053246-clearing-out-the-previous-program-on-a-micro-bit. But there doesnot seem to be any link to the two required files; does any of you have the mentioned auto_on.cfg and erase.cfg files ?
• we even tried that same suspicious HEX on a 2nd working Microbit ... it now seems bricked as well.

Here are the symptoms:

• Yellow LED flashes a couple of times then stop (on the firmware-upgraded Microbit); yellow flashes then remain solid (on the 2nd Microbit with no firmware update, version 0241 enclosed)
• Both Microbit are detected by Windows and appears as drive D:
• Able to browse the content D: (DETAILS.TXT, MICROBIT.HTM, etc)
• Fails whenever an HEX download is attempted.

This suspicious HEX was made from the MICROBIT online 'block' interface on Chrome 63.0.3239.84 with no ad-blocker and only the 'default' add-ons, antivirus sis BitDefender free 10.21.1109, basic windows firewall. That very same setup had issued operational HEX just a few minutes before this suspicious HEX.
The initial download of the suspicious HEX was made under Microbit Firmware Interface 0241.
They had both the USB and 3V connected... not sure if they are supposed to do it this way but they claimed that's how they do it at school and had done it that way numerous time at home before.

I now have two 'bricked' Microbit. Both were previously successfully updated using that very same setup. I don't think it is some power failure during transfer or ESD zap or ...
It really seems to be a corrupted HEX.

Do any of you have any tips on how to recover them ?
If more info would be useful, please shoot.

removed.hex.zip While we work out what's going on I've removed the hex in question. Though I can't reproduce the failure on the setup I have right now, if this hex does cause trouble for other users I'd like to limit exposure to it until further investigation is complete. I'm very happy to share it with other developers or debuggers -- jaustin.
ASSERT.TXT
DETAILS.TXT