[Peer]

… I will play as often as I can over the weekend. And I will continue my own experiments / investigations of port b input-output behavior. There are some things I still do not understand. Could you shortly describe the “fix” you implemented to make version 45 faulty-tolerant?

Cheers and Encouragements,
Peer

[Peer]

… ah, shoot, sorry to hear about all that 🙁

Oh, boy, if I only knew any way to cheer you up? What about a virtual beer tonight? Or a large box of virtual chocolate bars?

It might be in vain, but here is an after-thought to my last post:

Your example uses the (first CLR, then later) INC in order to effectively put a 1 into PB0. What if the <something inbetween> causes a condition so that the INC reads a 1 on PB0? Not because of any programming error on your side, but due to some internal VIA glitch/age/read-from-output/whatsoever effect? Then the INC would put a 0 into PB0. Could that lead to a crash?

I am trying to put the blame on PB0 rather than PB6.

Under the assumption that an INC (which does a read-modify-store on the micro-cycle-level) simply does not (always?) work correctly in combination with a VIA port.

I hope you know what I mean. Sometimes I find it difficult to put my thoughts into exact words, especially when it comes to the low-level Vectrex stuff…

[Peer]

Ok, here is another sub-theory of the “weird VIA” kind:

It is closely related to the ones we have been pursuing recently, and it is directly related to your post about the INC instruction (http://vide.malban.de/1st-of-june-faulty-via-vs-inc).

Phenomenon + hypothesis:

Sometimes, PB6 does not have the value that was expected, and as a consequence an unintended bank switch happens.

Potential causes:

a) A faulty VIA: reading PB6 does not yield the value that was previously stored.

b) A glitchy VIA: only every once in a while, for no apparent reasons, reading PB6 does not yield the value that was previously stored.

c) A normal VIA: reading PB6 gives the correct value, but some not yet identified cause has changed this value and so it differs from the one that is expected (i.e. the one that the VB code has previously stored).

My train of thoughts is following c) now.

I read the Rockwell 6522 data sheet. The upper section of page 4, called “Port A and Port B Operation”, among other things describes what is supposed to happen if PB6 is read while being configured as output. Basically it states that you get the “value that was last stored in PB6”, whatever “stored” does mean in this context (A store while being configured as output?, I have to admit, that I do not understand everything to the last detail here).

Malban’s post about the INC instruction pictures the following situation:

CLR VIA_port_b
<something inbetween>
INC VIA_port_b

The micro-code of the INC instruction reads and then writes to port b. It should read an 8-bit zero, and then increment that zero to an 8-bit one. Thus, PB6 should have value 0. What now, if that <something inbetween> alters the value of port b, so that the INC does not read an 8-bit zero? I am not talking about some hardware glitches or a faulty VIA, I am looking for some perfectly valid reason for this.

The Vectrex Programmers Manual Part 1 describes port b as follows:

Port ‘B’ Data – Vectrex Hardware Control [CNTRL]
Bit Description
0 Sample / Hold strobe
1 Analog multiplexer select, bit 0
2 Analog multiplexer select, bit 1
3 Sound generator, term ‘BC1’
4 Sound generator, term ‘DBIR’
5 Analog compare term (Input to 6522)
6 –
7 Integrator ramp

Some of those are possibly configured as inputs, the others as outputs. Now, when clearing port b, and then later on reading from port b, do we really get a zero? Or do we get “some” 8-bit value which is possibly quite different, depending on the whole configuration of the pins, and on the instruction sequence executed before.

If the INC reads a pattern like xx111111 (and right now I do not see any strict proof that the INC really reads the zero that was stored by the CLR), then the increment would change the value of PB6. So would a DEC if it reads a x0000000 pattern.

I do not fully understand PB3-5, but maybe some sound playing can alter the read values of all port b bits?

Yes, this is a shot in the dark, but the patterns mentions above would probably be corner case situations which might occur only under very rare circumstances while playing VB.

I do not know if this is possible, but can the <something inbetween> code part be interrupted? By a processor interrupt I mean? Could there be other instructions, maybe using addressing modes which, as a side effect, change the value of port b (post inc or dec)?

Does this make any sense?

Cheers,
Peer

[Peer]

For easyness I translated the VIA test to “C” (and you will see – it really IS a complicated test) :-).

Lol, yes, that is something even I understand 😉 Thanks for the code, I will start testing tonight.

And just to be sure: Does this C code fire for the infamous broken VIA I sent you?

Cheers,
Peer

[Peer]

As of now – the “tester” only tests once.
I can certainly do such a repeated “tester” – thing is… it just be, that there are very weird prerequisite, so that VIA might behave differently. Some IRQ at a specific falling edge of a cycle… Timer T1/T2 overflow… or whatever.

So while very interesting to do such testing – it might still be that such a tester does not show a useful result. Since I can not guarantee, that all “cases” are really tried out…

Yes, I know, it is just a shot in the dark. But, following your VIA line of thoughts …

Conjecture: The “Bug” might not be due to some “logical” and deterministic code-related situation, but due to some old and aged piece of hardware (some analog effects in some old capacitor, … ???) in combination with only certain types of VIAs. Thus, the bug appears only “randomly”, and possibly without any code-related cause. So if a deterministic ForeverWeirdTest, which just repeats the very same test over and over again, suddenly fires, then we know that we are screwed. If the test does not fire, we know nothing. Or maybe get a tiny bit of assurance that we are not dealing with such (age) effects.

If I were only half as bright as you are, and if I had your experience and knowledge, then I would try to write such a test on my own, and simply let it run. But alas, I am not 🙂

If your test basically consists of just one “subroutine” which can be called from any kind of code context, then you could send me the code, and I could rather quickly generate all sorts of context situations in C, and then just call your test code. Or call the test code within my past C game projects and just let them run. That would increase the number of code scenarios tested…

[Peer]

… very interesting thought. I was digging in a similar direction in my post on March 26. So, question:

The WeirdTest binary you wrote for checking the type of VIA, as far as I understand, does some sort of “test”. Does it do this “test” only once, and then come up with an answer, or is the “test” done repeatedly?

What if you created a “while(1) {test}” ForeverWeirdTest binary that just keeps on checking the VIA (no VB stuff)? If your theory is correct, that some VIAs behave weird only every once in a while, then shouldn’t this be a way to chase such behavior?

Cheers,
Peer

[Peer]

Sorry to hear that the bug still hides in the shadows. I will run the mad bankswitching over the next days and report back. And on the risk of repeating myself, let us know if there is anything else we can do.

Cheers,
Peer

Edit: Hehe, thought the link was pointing to a test binary. Have watched the video now. If you deem it useful, send me the binary, and I will gladly test it on my consoles.

[Peer]

Hypothesis: Bankswitching

Would it make sense, if you created several binaries which do not contain the (full) VB code, but which just do some heavy load bankswitching? No real game stuff, just bankswitching related things.

Things which you consider as “this is not the normal or average bankswitching use case, but it should work”. The more different scenarios, the better.

We then run those binaries on our consoles and see what happens. If one of them produces a crash, this might give a hint to the corner case situation in VB.

Cheers,
Peer

[Peer]

Ok, here is my next attempt:

We know that there are different models of VIA chips out there. And I still remember the infamous faulty VIA I sent you. If I recall corectly, the VIA is used for bank switching and the differences between the VIA models are timing related and you found a way to deal with all variations in your code. I also recall that you wrote some test programm which identifies the type of VIA present in a console. So, might it be worth to check if all the consoles on which the current bug was observed have the same type of VIA? And all of us who did not observe the bug could check if we have different types?

And here are more questions:

– Do all known Vectrex consoles have exactly the same type of 6809 CPU? I know that there are also variations of 6809 processors, but I did not find out if different ones were used in the production runs of Vectrex consoles.

– Does your assembly code use some of the non-official 6809 instructions (sometimes also called undocumented or illegal instructions, you know what I mean)?

– RAM chips have setup/hold timing restrictions. Normally the hardware design of the whole system ensures that these are respected. Is there possibly any low-level timing-critical stuff that might violate them? (e.g. new value not yet stable in RAM cell when next instruction relies on it?) Are there possibly different types of RAM chips out there in Vectrex consoles which have timing differences which under normal circumstances do not matter, but come into play here?

Cheers,
Peer

[Peer]

… darn, finding a way to help you is hard 😉

This will not stop me from posting more wild guesses and other crazy things! Is there really nothing I/we can do to help you?

And the offer still stands. Also if you have other ideas of how to put the server to good use.

Cheers,
Peer

EDIT: Cool, the overlays look great!

[Peer]

… do you ever sleep? 😉

Next question regarding demo mode: Computers run their programs deterministically. Unless there is some users input or some true randomization, the output will always be exactly the same. How do you ensure that each demo-mode-run behaves differently? Are the RNGs seeded differently each time, depending on some user input or hardware timer? Just to prevent that each of us is just experiencing exactly the same game behavior when testing in demo mode (i.e. a behaviour that just does not activate the bug).

EDIT:

– What about more “demo modes” to incite more corner case behavior? E.g. a mode with constant maximum auto fire (lots of bullets permanently active)? Or a mode with always-dropping-bonus item? Or only-money-dropping? …

– This might not be feasible: massive parallelization, 8+ Vides running in parallel and doing different demos runs for a week. I have a server in my lab that has 8 CPUs, each of them having 4 cores. There is no easy way to give you access to that server, and it would surely require some additional setup of the Linux system, but if you are interested, let me know, and I will try to make this possible.

[Peer]

Three more questions:

– Does VB use any of the “original” BIOS/RUM routines? If so, which?
– Does VB use any of the ROM data (located in the BIOS or located in the MineStorm code)? If so, which?
– The bug occured on various real consoles. Do all of these console have the same BIOS version, or do we know that the bug occured with different BIOS versions?

Cheers,
Peer

[Peer]

Some more wild guesses. I am sure you most likely already thought about this, but I can’t think of any other way to help you right now.

I know that VB code is not “normal”, meaning it uses all sorts of sophisticated assembly language tricks. Under “normal” circumstances, with a bug situation like this (indeterministic, not reproducable), I would think of memory corruption and/or stack corruption. If I recall correctly, then you are also (ab)using the stack register in other ways, but anyway. I am just brainstorming. So what about:

– stack overrunning heap, and thus corrupting ram data

– stack corruption, some instructions like jsr or rts which also implicitly use the stack interfering with your using of the stack register for other purposes

– corrupted/abused stack followed by an rts will corrupt the program counter

– if you are storing/handling the game objects in some sort of linked list, what about list corruption? Maybe in combination or as a consequence of one of the above?

Another crazy idea: if we assume the bug to be purely programming-logic related (no hardware issue, no timing issue), then maybe Vide could be able to re-simulate it? Meaning: let Vide run the binary for hours, maybe in some special, yet to be implemented autonomous playing mode, and see if the bug occurs? Ok, you can see how highly I think of Vide 😉

Let me know if I can help with any of this, or in other ways…

Cheers,
Peer

[Peer]

Blind guess:

You said a crash can happen basically anywhere, and before that you listed several situations. Is there anything all those situations have in common? One thing crossed my mind: sound. I do not know anything about your sound engine. Maybe some bug occuring in some corner case situation if some new sound is starting while another sound is still playing?

This is just a wild guess, but I was thinking about what those situations could possibly have in common…

[Peer]

Ok, as far as I understand, the crash occurred on different consoles. Did it also occur with other testers, using other eproms? Or was it always you using the same eprom? Just to rule out some hardware glitch of your eprom?

[Author]

Posts