General workings of Joker. These are mostly internal details you shouldn't need just to
use Joker. However, they're nice to know anyway, so better stay with me.


1.0	USING JOKER:
	
	1.1	SYNTAX OF JOKER:
		
		To call Joker, enter
		
			Joker <fileName> [-u [on|off]] [-v [on|off]] [-f [on|off]]
		
		at the command line prompt. The characters beginning with a dash are
		options Joker lets you specify. If you specify an option but do not
		specify whether you want to turn it on or off, they will default to
		on. If you omit any of the options entirely, Joker will use its
		defaults, which currently is to provide the same behaviour as
		HyperTalk, all options on. Here is a list of what all options do:
		
			-u		Allow unquoted string literals.
			-v		Use variables to handle unquoted literals so they
					can be reinterpreted if they are loop conditions.
			-f		Distinguish between function and command handlers.
			-i		Display version information and copyright
		
		Some ports of Joker may be designed to run from a GUI instead of
		from a command-line prompt. See the documentation specific to the
		platform you are using to find out about any differences from the
		procedure described above.


2.0	HOW JOKER EXECUTES ONE SCRIPT:

	2.1	TOKENIZING:
	
		The class HyperTalk loops over the text string passed in and parses it for tokens.
		Every token is added to a list of tokens. This is where quoted string literals
		and numeric values are also turned into string and number tokens of the appropriate
		value, and this is where comments are skipped.
		
		That is, tokenization in Joker is a lossy process where the instructions are
		separated from other data contained in the script.
	
	2.2	PARSING:
	
		The class HyperTalk iterates through the list of tokens and generates HyperInstr
		instances for every instruction necessary to execute the script described by the
		tokens. This is where handler calls are turned into the appropriate push/JSR
		instructions (though not into processor-native assembly, but rather into an
		intermediate format specific to Joker) and expressions are broken up into several
		step-by-step instructions to perform the calculations. This is also where commands
		and control structures are analyzed for syntax compliance.
		
		Stack space needed for local variables and temporaries is determined at this step.
		
		2.2.1	EXPRESSION PARSING:
				
				The main action taking place is grouping values concatenated into one
				value using operators. Since this is done by sequentially iterating
				through the token stream, special actions have to be taken to adhere
				to mathematic standards like operator precedence. Every operator is
				assigned an own precedence value, where { +, - } < { *, /, div, mod }
				< { ^ }.
				
				The first value is transferred to the accumulator, then further
				operations are performed on that. An add instruction takes the form
				
					Accu + a
				
				which translates to the expression
					
					ADD( accu, a ) --> accu.
				
				When this is followed by an operator of higher precedence, e.g.
				
					a * b
				
				the previous expression's RHS value is hijacked and replaced with a
				temporary, into which the result of the multiplication instruction is
				stored. That is, the addition instruction above is replaced through two
				instructions
				
					MUL( a, b )			--> Temp1
					ADD( accu, Temp1 )	--> accu
				
				If now another action of the same precedence as the multiplication is
				encountered, it will again hijack the addition's second value just like
				the multiplication did, to yield e.g.
				
					MUL( a, b )			--> Temp1
					DIV( Temp1, c )		--> Temp2
					ADD( accu, Temp2 )	--> accu
				
				And this is how it basically goes on.
		
		2.2.2	PARSING MESSAGE HANDLER CALLS:
		
				When an unknown identifier is encountered at the start of a line, it is
				presumed to be the name of a message handler to be called. This handler
				name is followed by an arbitrary list of values, which are separated
				from each other using the comma identifier.
				
				Each of these values will be used as one parameter. Parameters are passed
				on the stack; that is, after a value has been parsed into the Accu, it is
				pushed onto the stack with the instruction
				
					PUS( Accu )
				
				Since a handler may receive an arbitrary number of parameters, the number
				of parameters is pushed onto the stack also, immediately followed by the
				JSR (=Jump to Subroutine) instruction. That is
				
					PUS( <param count> )
					JSR( <handler name> ) --> [the result]
				
				At this time the parser also declares the variable "the result" and
				specifies it as the destination of the JSR instruction's result.
		
		2.2.3	PARSING FUNCTION CALLS:
		
				A function call is conceptually the same as a handler call, however
				it is detected whenever an unknown identifier is encountered that is
				followed by an opening bracket. The parameters to a function call are
				enclosed in these brackets, or if no parameters are passed the
				brackets are specified without any value between them.
				
				The only difference between function and message handlers where the
				compiled output is concerned is that the result of a function call's
				JSR instruction is stored inside a temporary variable that is inserted
				into the expression the function call is part of, instead of in the
				variable "the result".
		
		2.2.4	CALLS TO BUILT-IN FUNCTIONS:
				
				When a built-in function that takes a single parameter whose instruction
				has been registered with the parser's OneArgFunction list is called directly
				bypassing the message hierarchy by using property syntax ("the numtochar
				of 2"), an instruction is immediately generated.
				
				Traditional function calls are passed up the message hierarchy and are
				executed only if they haven't been intercepted by any user-defined
				functions. In that case, if no more than four parametrs have been passed
				to the function (which is the limit of a TalkInstruction), they are
				removed from the stack and stored in a faked TalkInstruction which is
				passed to the appropriate TalkInstructionProc. If more than four
				parameters are passed, the param count is set to a special value and
				the parameters are passed on the stack.
		
		2.2.5	PARSING BUILT-IN COMMANDS:
				
				When an identifier matching the name of a built-in command is encountered
				at the beginning of a line, Joker looks up the corresponding command's
				TalkCommandEntry in its BuiltInCommand list. A TalkCommandEntry contains
				the ProcPtr for the instruction to generate, and a table mapping
				identifiers to the instruction's parameters. This table is used to parse
				the instruction in a more flexible way.
				
				Note: Since built-in commands are identified by their first token, commands
				that begin with the same token but generate different instructions must be
				manually coded into Joker.
		
		2.2.6	PARSING CHUNKS:
				
				When a chunk type is encountered when parsing for a value, several
				instructions are generated. First the range of the chunk is parsed from
				the script as expressions. Then the value to get the chunk from is
				parsed.
				
				This information is passed as parameters to the HyperGetChunkOffs
				instruction that is then generated, along with a constant identifying
				the chunk type. This instruction will at runtime parse the value to
				get the chunk from and determine its range. This range is stored in
				two thread-global variables.
				
				After the HyperGetChunkOffs instruction follows a HyperGetChunkRef
				instruction that generates a reference to the target value restricted
				to the chunk's range. This allows both setting and getting a chunk.
				
				The separation of chunk parsing into two instructions is intended to
				make nested chunks more effective. If a chunk of a chunk is
				parsed, it can simply remove the last HyperGetChunkRef instruction and
				then insert its HyperGetChunkOffs instruction that will take into
				account the chunk range set by the previous HyperGetChunkOffs
				instruction. This way, the chunk's range is reduced to its actual
				range and only then it is retrieved, thus reducing the need to copy
				and move around data unnecessarily and making it possible to also
				set nested chunks.
				
		2.2.7	PARSING ENTRIES:
				
				Entries are Joker's concept of arrays and are indexed using hashed
				text strings. Every variable may contain either a value (text, number)
				or several entries, which are in turn just variables, thus making
				multi-dimensional arrays a snap.
				
				Whenever the "entry" identifier is encountered, the class HyperTalk
				parses for an entry "key" (the hashed string) and a value from which
				to get the entry. Then it generates a HyperGetEntry instruction which
				uses that information to return a reference to the entry's value.
				This reference is then used to retrieve or modify the entry.
		
		2.2.8	GLOBALS:
				
				Declarations of global variables are parsed into a HyperDeclareGlobal
				instruction. This instruction takes a local variable with the name of
				the global and makes it a reference to the global's value. This
				instruction also creates the global if there is none of that name yet.
		
		2.2.9	DEBUGGING AID:
				
				To aid in debugging and error reporting, the class HyperTalk inserts
				a HyperDebuggerLineInstruction after parsing one line. This instruction
				gets the line number (howmany-eth line in the handler) as its argument.
				Usually this instruction will be a no-op, but it can be changed to
				put the current line number into a global or thread-global variable to
				aid in reporting errors or it can be changed to make the debugger
				highlight the appropriate line in the script when tracing.
				
				Furthermore, using various preprocessor constants, a simple mini-debugger
				that allows stepping through scripts at instruction level and other helpful
				tidbits may be activated. This mini-debugger also allows dumping the stack's
				contents etc.
				
	
	2.3	EXECUTION:
		
		The class HyperTalk iterates through the list of instructions and executes them.
		This is also where a stack (class ValueStack) is generated to be used for local
		variables and temporaries.
		
		2.3.1	RUNNING A HANDLER:
				
				Whenever a JSR instruction is encountered it calls the current script's
				Run() method with the handler name as its parameter. The parameters have
				already been pushed on the stack using Push-Instructions preceding the JSR
				instruction, as well as the number of parameters. The Run() method now
				looks up a handler of specified name and enlarges the stack to make space
				for the number of variable slots the called handler needs. Also, the base
				pointer of the stack is moved to point at the start of the variables. Then
				the instructions of the called handler are iterated over and executed. When
				a handler call is finished, the stack is made smaller by the combined size
				of the variables and the parameters, and the base pointer is restored to
				its previous location.
				
				When a message is passed on, only the local variables used by the current
				handler are removed from the stack, but the parameters stay the way they
				were before the Run() method was called. Then, the stack is simply passed
				on to the next object's Run() method which now runs as if it had been
				called directly.
		
		2.3.2	PARAMETERS:
				
				The base pointer is set to point at the start of a handler's local variables
				when it's called, and the last thing that is pushed on the stack before any
				handler is called are its parameters followed by the parameter count.
				Thus, whenever a parameter to a handler or function needs to be retrieved,
				Joker first looks at the stack entry immediately before the base pointer and
				checks whether enough parameters have been passed to retrieve the desired
				one. If there are too few parameters, Joker simply returns an empty string.
				Otherwise, Joker subtracts the parameter count from the base pointer and
				then adds the requested parameter to that to calculate the address at which
				the parameter passed lies and returns its value.
		
		2.3.3	RETURN VALUES:
		
				The result of a handler call is pushed onto the stack and is retrieved
				when restoring the caller's context after disposing the parameters and
				local variables. It is then assigned to the memory location specified
				in the JSR instruction's result.
				
				Note that the pushing onto the stack of the return value is done at the
				same time that the space for the local variables is created on the stack.
				An empty string is pushed onto the stack which the "return" command then
				changes to whatever value the handler returns.
		
		2.3.4	PASSING A HANDLER CALL:
				
				When the "pass" command is encountered, execution of the current handler
				is aborted and the stack used by this handler call is freed, however the
				parameters are left on the stack and then passed to the next object up
				the hierarchy which can use them like it had directly received them.
				
				
				
(c) Copyright 1999-2002 by M. Uli Kusterer, all rights reserved.
				
				