perl help needed

got any perl boffins, trying to get a bootloader working, part of it uses a perl script to read an intel hex file and turn it into a binary data file, its supposed to chop off all data from the hex file that is < address 0x800 .. instead I am getting a data file with 0x800 'ff's tacked on the front. using strawberry perl, its likely my compiler is chucking out a different hex file with all the addresses included instead of just 0x800 up where the program lives..

Code:

#!perl -w
 
 use strict;
 my $highlimit = 0xC000;
 my $lowlimit = 0x800;
 my $base_address = 0;
 my $current_address = 0;
 my %data;
 
 my $output = shift @ARGV;
 
 open(OUTPUT,">$output");
 binmode OUTPUT;
 
 while ()
 {
        if (/^:02000004(....)..\s*$/ )
        {
                $base_address = hex($1);
        }
        elsif (/^:..(....)00(.*)..\s*$/)
        {
                my $current_address = $base_address + hex($1);
                my $data = $2;
                my @data = $data =~ /(..)/g;
                foreach (@data)
                {
                        $data{$current_address++} = hex($_);
                }
        }
 }
 
 foreach (sort {$a$b} (keys (%data)))
 {
        if ($_ < $lowlimit or $_ >= $highlimit)
        {
                delete $data{$_};
        }
 }
 
 
 for (my $i = $lowlimit; $i < $highlimit; ++$i)
 {
        if (not defined ($data{$i}))
        {
                $data{$i} = 0xFF;
        }
 }
 
 foreach (sort {$a$b} (keys (%data)))
 {
        printf ("%4X %2X\n",$_,$data{$_});
        print OUTPUT chr($data{$_});
        # print OUTPUT chr($_ % 256); 
 }
 close OUTPUT; 
 

i can explain hex files and show you what i am using if that helps. all i have managed to work out so far is I unremmed out the first printf in the output loop, but it thrashes so much data to the console its pointless..

i have not examined the hex file thoroughly yet .. i am not sure if has all the 0 to 0x800 full of 0xff or not
so could be the delete loop is .comlty or the fill with 0xff bit .. not sure.

 

... deleted text

 

the point of this is converting an existing 18f encrypted bootloader to a 16f device, got most of it converted across except the actual flash writes now, the xmodem is now working, stupid PICC needs type defs on simple byte comparisons or it converts them to ints ... grr.

if i can get the bootloader working then we should be able to upgrade the software in the little ecig display units if i make any changes and anyone cares..

at the moment, all my debuugging for the xmodem has to be displayed on the 3 digit 7 segment display as the terminal window is busy doing the xmodem and will not print debug prints

 

Duno!! ----------

 

turns out the output data file is far too long as well.

its no biggie really, i can manually cut the front and end of the file off with notepad++ with a hex editor add on, but i think i want to port the encryption to perl as well..

bootloader is _almost_ working..

 

You lost me just after "we got any perl boffins" mate, sorry.

 

Yup, me as well!!

 

well the bootloader part is working 100% .. now to get the offset code to work. i can program the data file in perfectly, disable the bootloader and the thing resets 30 times a second. the tools are being a pita, debugging with trace no good.
must be bedtime ..

 

Howdy, I can help with Perl stuff if you still need a hand Stoney

Can't spot anything obvious there that would .comse your issue, could you swing me an example of an input file that doesn't get processed correctly?

 

well its a bit strange .. I have been reading some perl docs and sort of getting some debugging going.
if I try a smallish file it works perfectly, a larger one .. not so.
I have moved onto another bootloader at the moment that does not need the perl script, turns out .. the whole thing is currently too hard. the compiler is completely ignoring most of the linker commands I need to get both bootloaders and the target program in the right areas, its simply not working and I rate it down my priority list right now .. I have wasted far too much time on the damn bootloaders recently


that being said .. I would still like to know whats going on, I may yet want to revisit this ..
heres a small example of a hex file offset to 0x800 hex..
it should create a dat file that starts with 3188 or 8831 (cant remember if its high byte first or last)

Code:

:0410000088313728D4
:10100800883120007F08F4000B1D0B280C2833289A
:101018008B1E0F28102833280130F0070030F13DCF
:1010280000307102653003197002031C1C281D284A
:101038003128F001F1010130F3007308F2072200B2
:101048000E18272828282A280E142C280E102C2899
:101058007208613E23009A0031280B113328740866
:101068002000FF000900880188316D28EF30210039
:101078009900DB308C00E8308E00103023008E00A1
:1010880008308C0024309E0090309D0008309F006E
:10109800CF309B0078309A0022008C168E140C15E5
:1010A800210095128B168B1706307202031C5D28DF
:1010B8005E285828F20122000E1D6328642866283D
:1010C8000E1158280E15582858285828582888319D
:0E10D8003728F001F101F201200088313A289A
:00000001FF

 

yep, just tried it .. 0x800 FF's .. then the 88 31 hex ..
viewing in notepad++ with hex viewer installed.

 

and now i know a bit more I can see the problem with that one .. the
02000004 is missing from that hex file..

 

ok same deal with this one .. had to 'extend address 0 in hex file' in linker options..

Code:

:020000040000FA
:0410000088313728D4
:10100800883120007F08F4000B1D0B280C2833289A
:101018008B1E0F28102833280130F0070030F13DCF
:1010280000307102653003197002031C1C281D284A
:101038003128F001F1010130F3007308F2072200B2
:101048000E18272828282A280E142C280E102C2899
:101058007208613E23009A0031280B113328740866
:101068002000FF000900880188316D28EF30210039
:101078009900DB308C00E8308E00103023008E00A1
:1010880008308C0024309E0090309D0008309F006E
:10109800CF309B0078309A0022008C168E140C15E5
:1010A800210095128B168B1706307202031C5D28DF
:1010B8005E285828F20122000E1D6328642866283D
:1010C8000E1158280E15582858285828582888319D
:0E10D8003728F001F101F201200088313A289A
:00000001FF

 

OK, yep the first hex file you posted had no base address, so all of the data would've been offset from 0. No biggie, just .comses perl to spit out a bunch of warnings due to uninitialised $base_address. Second hex file contains a base address, but that base address is 0 anyway! Heh...

So I've spruced up the script with some comments and debug output and stuff...

Here's the debug output from the first file:

Code:

[flat@mikoshi vapeboot]$ ./build.pl output.bin input1.hex 
No base address found!
De.comlting to 0x0000
Next address: 0x1000
Next address: 0x1008
Next address: 0x1018
Next address: 0x1028
Next address: 0x1038
Next address: 0x1048
Next address: 0x1058
Next address: 0x1068
Next address: 0x1078
Next address: 0x1088
Next address: 0x1098
Next address: 0x10A8
Next address: 0x10B8
Next address: 0x10C8
Next address: 0x10D8
Trimmed 0 out-of-bound addresses
Padded between 0x0800 and 0x1000 (0x0800 bytes)
Padded between 0x1004 and 0x1008 (0x0004 bytes)

Second file:

Code:

[flat@mikoshi vapeboot]$ ./build.pl output.bin input2.hex 
Base address: 0x0000
Next address: 0x1000
Next address: 0x1008
Next address: 0x1018
Next address: 0x1028
Next address: 0x1038
Next address: 0x1048
Next address: 0x1058
Next address: 0x1068
Next address: 0x1078
Next address: 0x1088
Next address: 0x1098
Next address: 0x10A8
Next address: 0x10B8
Next address: 0x10C8
Next address: 0x10D8
Trimmed 0 out-of-bound addresses
Padded between 0x0800 and 0x1000 (0x0800 bytes)
Padded between 0x1004 and 0x1008 (0x0004 bytes)

Note that 0x0800 becomes 0x0000 when the file is written, so we're definitely padding the first 0x0800 bytes as you say. Seems that the first address with data is 0x1000, so that makes sense... not sure what's required by the unit you're flashing, maybe we should just trim until the first data address (or hard-code 0x1000 as the low limit)?

Oh yeah, here's teh codez:

Code:

#!/usr/bin/env perl

use strict;
use warnings;

my $lolimit = 0x800;
my $hilimit = 0xC000;

my $usage = sprintf "usage: %s output.bin input.hex [input.hex ...]

Parses the input hex files, trims the data outside 0x%04X and 0x%04X,
fills any holes between those addresses with 0xFF, and then writes the
output binary.
", $0, $lolimit, $hilimit;

die $usage if @ARGV < 2;

my $output = shift @ARGV;

my $base_address    = undef;
my $current_address = undef;
my %data            = ();

open OUTPUT, '>', $output or die "Couldn't open '$output': $!";
binmode OUTPUT;

while () {
    if (/^:02000004(....)..\s*$/ ) {
        # :02000004 0000 FA

        $base_address = hex($1);
        printf "Base address: 0x%04X\n", $base_address;
    }
    elsif (/^:..(....)00 (.*)                             ..\s*$/x) {
        #    :04 1000 00 88313728                         D4
        #    :10 1008 00 883120007F08F4000B1D0B280C283328 9A

        # Catch cases where the file has no base address
        unless (defined $base_address) {
            print "No base address found!\n";
            print "De.comlting to 0x0000\n";
            $base_address = 0;
        }

        # Determine the address for this chunk of data
        my $current_address = $base_address + hex($1);
        printf "Next address: 0x%04X\n", $current_address;

        # Store the bytes against their address
        my @bytes = $2 =~ /(..)/g;
        $data{$current_address++} = hex($_) for @bytes;
    }
}

my $before = scalar keys %data;

for (sort {$a  $b} keys %data) {
    # Cull any data outside of lolimit and hilimit
    delete $data{$_} if $_ < $lolimit or $_ >= $hilimit;
}

my $after = scalar keys %data;

printf "Trimmed %d out-of-bound addresses\n", $before - $after;

my $pad_start = undef;

# NOTE: It would be more efficient to write the output file in this loop,
#       rather than filling up our hash only to loop over it once more
for (my $i = $lolimit; $i < $hilimit; ++$i) {
    if (not defined $data{$i}) {
        # Pad any unset addresses between lolimit and hilimit
        $data{$i} = 0xFF;
        # Record the start of a padding section
        $pad_start = $i unless defined $pad_start;
    }
    elsif (defined $pad_start) {
        # Record the end of a padding section
        printf "Padded between 0x%04X and 0x%04X (0x%04X bytes)\n",
            $pad_start, $i, $i - $pad_start;
        $pad_start = undef;
    }
}

# Write the output file
for (sort {$a  $b} keys %data) {
    # printf "%4X %2X\n", $_, $data{$_};
    print OUTPUT chr($data{$_});
}

close OUTPUT;

 

interesting, the first address with data is actually 0x800.

the idea behind this loader is it resides in the low 0x800 .. files to be loaded are compiled with an offset of 0x800. this puts the reset vector at 0x800 and interrupt vector at 0x804. loader exits with a jump to 0x800 and the real interrupt vector contains a jump to 0x804 .. which is actually problematic and the reason I will likely not use this loader .. yet.

dat file is sent to the loader via xmodem with no addresses, loader takes the data and programs in from 0x800 to top of memory, 0x1ffff in actuality.

this is the wombat bootloader, the original was for a 18f and I modded the loader for hitech c instead of pic c18.

what i may yet do, relocate the loader to top of rom, load programs low, protect the reset vector jump to loader and relocate what the dat file attempts to load as a reset vector to program in as the loaders exit. interrupt vector stays at 0x0004 then.

but hitechc is jst not playing proper, reset vectors blowing out to 5 bytes, ignoring absolute addresses, being insanely tricky with psects .. not relocating strings as requested, very annoying.

 

efficiency is not a problem btw, it runs in a few seconds and is only required once per build.

 

Hrrm, well it definitely looks like the hex is offset at 0x1000, you can spot it right in the hex on the second line (check the regex to see which chars contain the offset). To trim off all the padding at the start, simple solution is to set $lolimit=0x1000 near the start of the file. A better solution might be to remove the lolimit altogether and just write from the first data address, though that could go wrong if you need some padding at the start (guessing 0xff is a nop here?)

Not too familiar with PICs, though have stuffed around with some AVRs in the past. You said the loader was for an 18f, what are you using now? Compiler putting stuff out-of-bounds suggests that maybe it's set up for the wrong target?

The efficiency stab was just me being pedantic don't mind me! I figured it didn't matter much.

 

ah yes .. pics are a 14 bit device, so program counter 0x800 is byte 1000h and 1001h

i was viewing the hex in the program memory tab, which has it at 0x800

 

target is now a 16f1825, target is set correctly for the complier/linker .. pdf manual open in other window..
compiler commands are for the command line, compiler is now under a IDE and it seems not perfectly integrated.