Al, Thanks for your response. We've been focusing on the ISRs pretty heavily for the last 2 weeks or so. I have created instrumentation that marks the entry and exit points of each C interrupt service routine. I struggled with the assembly language portion of the interrupt handling as I wanted to instrument as close to the entry and exit as possible. It was unable to come up with anything that worked. So far, in all of the resets that I have data on, I have been unable to find a spot where I can determine that normal execution has ceased as a result of an interrupt. My instrumentation is pretty limited on what information I have on the normal main loop of execution. Most of the instrumentation is in the ISRs. Right now, I've got a unit running on the emulator with hardware breakpoints set to catch execution outside of expected address space and also writes to either of the main sections where the executable resides. Thanks to everyone who has made a suggestion or asked a question. It really does help. Rob
Dredge Up ADSP-21161 RTI(DB) Thread
Started by ●May 23, 2006
Reply by ●May 24, 20062006-05-24
Reply by ●May 25, 20062006-05-25
Here is what we are using for interrupts: Timer - set to 50 usec period IRQ2 - ARINC Communications interrupt. Handles ( or is supposed to ) 4 receive channels and 2 transmit channels. The frequency of this interrupt ranges from less than 50 usec up to 400 usec with full bus traffic on 2 of the receivers ( like in the field ) and pretty heavy traffic on the 2 transmit channels ( normal operation ). SPORT0 - This interrupts at a 20msec rate. SPORT2 - This interrupts at a 20msec rate. We used to have the UART interrupt driven, but it was changed to polled a few months back. That was the original 'fix' for our resetting problem. We have also tried changing the SPORTS to being polled and we have been running for over a week without a reset. It looks like a reduction in interrupt load makes things better. I am still not sure it is 'fixed', though. Thanks again, Rob
Reply by ●May 26, 20062006-05-26
My latest build has the Interrupt Nesting disabled, and I have not had a reset in just over 14 hours. We've had runs like this before, so I'm not getting my hopes up. Is anyone else using the interrupt dispatching supplied by Analog Devices? Do you have interrupt nesting enabled? Have you had any problems like this? Thanks, Robert Allen
Reply by ●May 26, 20062006-05-26
"rallen911" <rallen_prsch911@yahoo.com> wrote in news:1148647430.291217.227460@j33g2000cwa.googlegroups.com:> My latest build has the Interrupt Nesting disabled, and I have not had > a reset in just over 14 hours. We've had runs like this before, so I'm > not getting my hopes up. > > > Is anyone else using the interrupt dispatching supplied by Analog > Devices? Do you have interrupt nesting enabled? Have you had any > problems like this? > > Thanks, > > Robert Allen > >This suggests that you have something in your interrupt routines that is causing the problem. My guess is that with nesting on, a higher priority interrupt is corrupting a lower priority interrupt. For example, Lets say both routines use r1. The higher priority interrupt must save a copy of r1 and restore this value on exit. This would also be true for values in the MR register. When interrupts are not nested (my normal preference), this can't happen. If you use secondary registers and DAGs and unnested interrupts, there should be no interference. Generally your program is constructed so that other resources that are not duplicated reside in either the foreground or background but not both. I tend to write short ISRs that fetch data and set semaphores. The foreground does the signal processing. The other typical approach is to do everything in the ISR and very little in the foreground. The latter approach gets harder to manage as programs get larger. Al -- Al Clark Danville Signal Processing, Inc. -------------------------------------------------------------------- Purveyors of Fine DSP Hardware and other Cool Stuff Available at http://www.danvillesignal.com
Reply by ●May 26, 20062006-05-26
Al Clark wrote:> "rallen911" <rallen_prsch911@yahoo.com> wrote in > news:1148647430.291217.227460@j33g2000cwa.googlegroups.com: > > >>My latest build has the Interrupt Nesting disabled, and I have not had >>a reset in just over 14 hours. We've had runs like this before, so I'm >>not getting my hopes up. >> >> >>Is anyone else using the interrupt dispatching supplied by Analog >>Devices? Do you have interrupt nesting enabled? Have you had any >>problems like this? >> >>Thanks, >> >>Robert Allen >> >> > > > This suggests that you have something in your interrupt routines that is > causing the problem. My guess is that with nesting on, a higher priority > interrupt is corrupting a lower priority interrupt. > > For example, > > Lets say both routines use r1. The higher priority interrupt must save a > copy of r1 and restore this value on exit. This would also be true for > values in the MR register. > > When interrupts are not nested (my normal preference), this can't happen. > If you use secondary registers and DAGs and unnested interrupts, there > should be no interference. Generally your program is constructed so that > other resources that are not duplicated reside in either the foreground > or background but not both. > > I tend to write short ISRs that fetch data and set semaphores. The > foreground does the signal processing. The other typical approach is to > do everything in the ISR and very little in the foreground. The latter > approach gets harder to manage as programs get larger.Al, what you wrote rang a bell. One fellow assured me that he was saving the internal states of interrupted interrupt routines, and he was -- in variables. Not separate variables for each IR, but common ones. Think what happens when an interrupting routine gets interrupted. All that stuff belongs on the stack. If there's not enough RAM for the stack, there's not enough RAM to store all the machine states. Jerry -- Engineering is the art of making what you want from things you can get. �����������������������������������������������������������������������
Reply by ●May 26, 20062006-05-26
Jerry Avins <jya@ieee.org> wrote in news:prSdnaBnVrtmgOrZnZ2dnUVZ_vydnZ2d@rcn.net:> Al Clark wrote: > >> "rallen911" <rallen_prsch911@yahoo.com> wrote in >> news:1148647430.291217.227460@j33g2000cwa.googlegroups.com: >> >> >>>My latest build has the Interrupt Nesting disabled, and I have not >>>had a reset in just over 14 hours. We've had runs like this before, >>>so I'm not getting my hopes up. >>> >>> >>>Is anyone else using the interrupt dispatching supplied by Analog >>>Devices? Do you have interrupt nesting enabled? Have you had any >>>problems like this? >>> >>>Thanks, >>> >>>Robert Allen >>> >>> >> >> >> This suggests that you have something in your interrupt routines that >> is causing the problem. My guess is that with nesting on, a higher >> priority interrupt is corrupting a lower priority interrupt. >> >> For example, >> >> Lets say both routines use r1. The higher priority interrupt must >> save a copy of r1 and restore this value on exit. This would also be >> true for values in the MR register. >> >> When interrupts are not nested (my normal preference), this can't >> happen. If you use secondary registers and DAGs and unnested >> interrupts, there should be no interference. Generally your program >> is constructed so that other resources that are not duplicated reside >> in either the foreground or background but not both. >> >> I tend to write short ISRs that fetch data and set semaphores. The >> foreground does the signal processing. The other typical approach is >> to do everything in the ISR and very little in the foreground. The >> latter approach gets harder to manage as programs get larger. > > Al, what you wrote rang a bell. One fellow assured me that he was > saving the internal states of interrupted interrupt routines, and he > was -- in variables. Not separate variables for each IR, but common > ones. Think what happens when an interrupting routine gets > interrupted. > > All that stuff belongs on the stack. If there's not enough RAM for the > stack, there's not enough RAM to store all the machine states. > > JerryYou rarely need a general purpose stack on a SHARC if you are 1. Not nesting your interrupts and 2. Using secondary DAGS (pointers) and registers and 3. writing assembly code The overhead can be very small: _typical_isr: PUSH STS; // Status Registers BIT SET MODE1 various secondary registers and dags; .... interrupt stuff POP STS; RTI; The overhead can be very small. Usually if you are nesting interrupts, you try to use different registers and DAGs for each interrupt so that they don't interfere. If you do need to save something, you only need to save the register in the higher priority ISR. I would write something like this: DM(r4_reg) = r4; ..... r4 = DM(r4_reg); RTI; If there are multiple interrupts (nested) DM(r4_reg_isr_whatever) = r4; // Use a more specific name ..... r4 = DM(r4_reg_isr_whatever); This is necessary to avoid the situation that Jerry mentioned. If you keep you interrupts short, you can usually avoid nesting, which makes the situation very easy and you probably can avoid the need for a stack. Stacks address the collision issue at the cost of more overhead. If you are using C isrs, the compiler will probably assume you need to save most everything, which should avoid collisions at the cost of additional overhead. In all cases, I would examine each isr carefully to make sure that you can account for all situations. Since your problem appears "random", I think this is the problem. I get very uneasy when problems are solved by moving things around without determining the problem in the first place. These solutions often come back to bite you later. -- Al Clark Danville Signal Processing, Inc. -------------------------------------------------------------------- Purveyors of Fine DSP Hardware and other Cool Stuff Available at http://www.danvillesignal.com
Reply by ●May 26, 20062006-05-26
I agree with everything you have said. Way back when we started the project ( 3 years ago ), I decided to use the interrupt dispatching / handling built into VDSP in order to save some time in getting going. I always planned on going back and replacing that with our own, more specific code. As you can probably guess, the time to do that never materialized and we've gone through final verification with that code. There wasn't anyone on the team that had written assembly for the ADSP, so nobody was really hot to do it. We ended up writing all of our ISRs in C, so we are relying on the VDSP library routines to save and restore everything each interrupt. We have nesting turned on normally, and I think that is what our big problem is. In 2 independent tests, we have isolated the problem to the interrupt handling. The first test has the interrupt nesting turned off, and it has run for over 24 hours without a reset. The other test disable interrupts in the sections where the VDSP code saves and restores the registers. We don't specifically change to use the secondary registers at any time, so maybe that could be how we isolate the interrupt routines. All of our code is in C, except the start-up and VDSP libraries that we use, and I don't think any of those alter which set of registers is being used. Do you know of anyone else who is using the built-in interrupt dispatching / handling?
Reply by ●May 31, 20062006-05-31
Al Clark wrote:> In all cases, I would examine each isr carefully to make sure that you > can account for all situations. Since your problem appears "random", I > think this is the problem. I get very uneasy when problems are solved by > moving things around without determining the problem in the first place. > These solutions often come back to bite you later. >One other thing to pay close attention to is the use of circular buffering. The ADI documentation says that the L4 and L12 registers are scratch (meaning they can be modified in a subroutine and not restored), but that's not exactly true. The compiler always assumes that these registers are 0, meaning linear addressing. If you EVER change an L register to non-zero, make sure you cannot be interrupted while that condition persists. An ISR could go off and use the corresponding I-reg to index something in memory and get an unintended modulus applied to it. I haven't looked at the interrupt dispatchers in a while, so they MAY have fixed this, but they may not have either. Something to look for. -- Jim Thomas Principal Applications Engineer Bittware, Inc jthomas@bittware.com http://www.bittware.com (603) 226-0404 x536 I thought I was wrong once, but I was mistaken.
