RFC: Application Package Format

Alcoholics Anonymous
Mensajes: 10
Registrado: 14 Nov 2016, 04:53

Re: RFC: Application Package Format

Mensaje por Alcoholics Anonymous » 07 Jul 2018, 08:33

It's been a while and I am about to implement a RIFF-like .exe format for the next. However, it is necessarily very next-centric because of the machine's feature set. The start-resume block might be a transferable idea that allows the program to run part way through the load. This would be for introductions, configuration, cut scenes, etc. Another idea is the obfuscation seed in the stored data; this is likely going to be a seed for a prng that generates bytes XORed with data after it is loaded. This is not a protection scheme but a means to hide plaintext as people on facebook asked that files not be readable in a text editor. Apparently the temptation to cheat in games is too great. Remark blocks could contain information like copyright, brief description and so on, that could be printed via the dot loader instead of loading the program. A special loading screen block is not defined yet but will be added. Having a separate loading screen block not only makes it easy to add one, but allows the dot loader to display it instead of running the program.

Most of the existing blocks are for loading into a logical memory space (one that is allocated from free memory pages) where the loader needs to create an allocation table and write it into the loaded program. Anyway because of the unique features of the next, the format is necessarily next focused.

Código: Seleccionar todo

**************************************************************************************

Header Format:

OFFSET   SIZE  DESCRIPTION

0        2     Characters 'NX' ('N' in byte 0, 'X' in byte 1)
2        2     File format version (should be 0)
4        2     Size, in bytes, of the data following these two bytes

6        1     Size of page allocation table (0 if none)
7        1     Size of extra pages allocation table (0 if none)
8        1     Size of divmmc allocation table (0 if none)

9        ?     Page allocation table (0xff = skip, 0xfe = allocate, other = error)
?        ?     Divmmc allocation table (0xff = skip, 0xfe = allocate, other = error)


Block Format:

OFFSET   SIZE  DESCRIPTION

0        2     2 characters describing block format
2        2     Size, in bytes, of the data following these two bytes
4        ?     Block data


**************************************************************************************



-- Remark

OFFSET   SIZE  DESCRIPTION

0        2     Characters '--'
2        2     Block format version (should be 0 for printable comment block)
4        2     Size, in bytes, of the data following these two bytes

6        ?     Optional data that may be skipped by loader





PL Page Load

OFFSET   SIZE  DESCRIPTION

0        2     Characters 'PL'
2        2     Block format version (should be 0)
4        2     Size, in bytes, of the data following these two bytes

6        2     Obfuscation seed (0 = none)

8        1     Page # to load to (range 0-223)
9        2     Offset into page (range 0-8191)
11       2     Number of bytes to load (can span several pages)

13       ?     The bytes to load into the page, length equal to value at offset 11




DL Divmmc Load

OFFSET   SIZE  DESCRIPTION

0        2     Characters 'DL'
2        2     Block format version (should be 0)
4        2     Size, in bytes, of the data following these two bytes

6        2     Obfuscation seed (0 = none)

8        1     Page # to load to (range 0-15)
9        2     Offset into page (range 0-8191)
11       2     Number of bytes to load (checked to remain within 8k divmmc page)

13       ?     The bytes to load into the divmmc page, length equal to value at offset 11



;;; LS Load Screen
;;;
;;; OFFSET   SIZE  DESCRIPTION
;;; 
;;; 0        2     Characters 'LS'
;;; 2        2     Size, in bytes, of the data following these two bytes
;;;
;;; Not specified yet
;;; Screen data already loaded, copy to destination for effects?
;;; Choose ula mode (spec, timex, lores) or layer 2
;;; set palette maybe different block to load nextreg state
;;; loading effect (eg border)



PB Pause Block

OFFSET   SIZE  DESCRIPTION

0        2     Characters 'PB'
2        2     Block format version (should be 0)
4        2     Size, in bytes, of the data following these two bytes (4)

6        2     Wait in milliseconds (0xffff = wait for no keypress)
8        2     Pause in milliseconds (interrupt with keypress and maybe with joystick fire)




CL Write command line

OFFSET   SIZE  DESCRIPTION

0        2     Characters 'CL'
2        2     Block format version (should be 0)
4        2     Size, in bytes, of the data following these two bytes (8)

6        2     Page # to write to
8        2     Offset into page (range 0-8191)

10       1     Command line type (0 = unprocessed esxdos, 1 = unprocessed nextos-esx, 2 = empty argc/argv, 3 = argc/argv)
11       1     Write direction (0 = forward else stack push)
12       2     Maximum length in bytes (can span pages)



WD Write current working directory

OFFSET   SIZE  DESCRIPTION

0        2     Characters 'WD'
2        2     Block format version (should be 0)
4        2     Size, in bytes, of the data following these two bytes (4)

6        2     Page # to write to
8        2     Offset into page (range 0-8191)




AT Write allocation table

OFFSET   SIZE  DESCRIPTION

0        2     Characters 'AT'
2        2     Block format version (should be 0)
4        2     Size, in bytes, of the data following these two bytes

6        2     Page # to write to
8        2     Offset into page (range 0-8191)

; list follows terminated by zero byte

?        1     1 = Write sysvar bank state
?        1     2 = Write initial mmu state
?        1     3 = Write page allocation table
?        1     4 = Write extra pages allocation table
?        1     5 = Write combined page and extra pages allocation table
?        1     6 = Write divmmc allocation table
?        1     0 = End of list (any other byte)




ST Start Block

OFFSET   SIZE  DESCRIPTION

0        2     Characters 'ST'
2        2     Block format version (should be 0)
4        2     Size, in bytes, of the data following these two bytes (13)

6        2     PC
8        2     SP (required if PC < 0x4000 and must be >= 0x4020)

10       1     Flags (bit 0 = 1 for DI, bit 7 = 1 to keep file open)
11       8     MMU state (mmu0...mmu7, 0xff = no change; ROM3 Banks 5,2,0 default)

; start with:
;
;   hl = argv / esxdos command line
;   bc = argc / esxdos command line length
;    e = file handle




SR Start-Resume

;;; Program runs with ".run" still in divmmc memory and returns to resume
;;; loading.  For introductions, cut-scenes, instructions, etc.

OFFSET   SIZE  DESCRIPTION

0        2     Characters 'SR'
2        2     Block format version (should be 0)
4        2     Size, in bytes, of the data following these two bytes (13)

6        2     PC
8        2     SP (required, return address on stack)

10       1     Flags (bit 0 = 1 for DI)
11       8     MMU state (mmu0...mmu7, 0xff = no change; ROM3 Banks 5,2,0 default; mmu0/1 ignored)
Matt also landed on the RIFF format idea and began this spec with the leading NX identifier block and a couple of blocks to load into physical memory and start. I've filled in the rest to do with logical memory mapping as I'm the only one working on that.

In the meantime, several big memory snapshot formats arrived. One is the big sna which is a standard 128k snapshot with data for more memory banks appended. This is compatible with esxdos. SNX is like a big sna but when it is loaded, the file handle for the SNX file is left open so the program can continue to load without having to know its filename (the big sna must have its filename hardcoded into its binary). These are mainly for development because one of the most important Next emulators is SNA only. Jim's big file format is called NEX and is memory state with an optional loading screen.

I did implement environment variables and temp files for esxdos and finished the .run dot command mentioned earlier.
https://github.com/z88dk/z88dk/tree/mas ... ommand/run

This dot command uses an environment PATH variable (the environment is just a text file holding "name = value" pairs) to find and load programs. It currently handles .tap, .sna, .snx, .z80, .p, .o, .nex. NextOS supplies the sna/snx/z80/p/o capability but I plan to build a version for esxdos as well using snapload so an esxdos version should be able to support tap, sna, z80 at least. For tap, I decided on exiting after the tap file is mounted with instructions to LOAD"" after the dot command completes.

Avatar de Usuario
chernandezba
Mensajes: 841
Registrado: 02 Oct 2015, 23:35

Re: RFC: Application Package Format

Mensaje por chernandezba » 09 Jul 2018, 14:16

Good idea. But I think there's a mistake on just using two bytes to tell the data lenght: you are limiting data blocks to 64kb.
Why not using 3 bytes, so having 16 MB data block maximum?
----

ZEsarUX
ZX Second-Emulator And Released for UniX
https://github.com/chernandezba/zesarux

Alcoholics Anonymous
Mensajes: 10
Registrado: 14 Nov 2016, 04:53

Re: RFC: Application Package Format

Mensaje por Alcoholics Anonymous » 10 Jul 2018, 03:27

chernandezba escribió: Why not using 3 bytes, so having 16 MB data block maximum?
Yes I am thinking about that too. I may make it 4 bytes instead as that will allow skipping over large sections of the file. The format allows unloaded data to be appended to the file for things that are loaded on demand at runtime and the SR block (start and resume) may want to embed a large unformatted data block that is skipped by the loader. There are people doing video that might need segments > 16MB.

I may remove the obfuscation as well and let the program / generating tool handle that.

Avatar de Usuario
chernandezba
Mensajes: 841
Registrado: 02 Oct 2015, 23:35

Re: RFC: Application Package Format

Mensaje por chernandezba » 10 Jul 2018, 09:58

Alcoholics Anonymous escribió:
chernandezba escribió: Why not using 3 bytes, so having 16 MB data block maximum?
Yes I am thinking about that too. I may make it 4 bytes instead as that will allow skipping over large sections of the file. The format allows unloaded data to be appended to the file for things that are loaded on demand at runtime and the SR block (start and resume) may want to embed a large unformatted data block that is skipped by the loader. There are people doing video that might need segments > 16MB.

I may remove the obfuscation as well and let the program / generating tool handle that.
Yes, 4 bytes would be even better.
On my ZEsarUX snapshot file format (.ZSF) I use 32 bit values for data block lenght, and 16 bit values for data type
----

ZEsarUX
ZX Second-Emulator And Released for UniX
https://github.com/chernandezba/zesarux

Avatar de Usuario
aowen
Mensajes: 178
Registrado: 07 Oct 2015, 13:32

Re: RFC: Application Package Format

Mensaje por aowen » 20 Jul 2018, 16:58

Alcoholics Anonymous escribió:It's been a while and I am about to implement a RIFF-like .exe format
Ah, you're reinventing SZX only in an incompatible format that won't work with emulation. OK.
obfuscation seed
Can't programmers be bothered to do their own obfuscation? Oh well.
Matt also landed on the RIFF format
Two incompatible SZX clones. Marvelous.
One is the big sna which is a standard 128k snapshot with data for more memory banks appended.
SNA is more or less the Mirage Microdriver format. 128K SNA snapshots are Pentagon snapshots. esxDOS may treat them as 128 snapshots, but that's not what they are.
SNX is like a big sna
How many times is it necessary to reinvent uncompressed SZX?
one of the most important Next emulators is SNA only.
Not supporting SZX in this day and age is inexcusable.
Jim's big file format is called NEX
How many ... I think I'll go put the kettle on.
I did implement environment variables and temp files for esxdos and finished the .run dot command mentioned earlier.
https://github.com/z88dk/z88dk/tree/mas ... ommand/run

This dot command uses an environment PATH variable (the environment is just a text file holding "name = value" pairs) to find and load programs. It currently handles .tap, .sna, .snx, .z80, .p, .o, .nex. NextOS supplies the sna/snx/z80/p/o capability but I plan to build a version for esxdos as well using snapload so an esxdos version should be able to support tap, sna, z80 at least. For tap, I decided on exiting after the tap file is mounted with instructions to LOAD"" after the dot command completes.
So it's .snapload with an environment PATH variable. I'll take a look at it and see if I can support it on UnoDOS 3 without breaking compatibility. An esxDOS version running on the ZX-Uno should be able to support .tap, .sna, .z80, .p and .o (assuming I remember correctly and those last two are ZX81 and ZX80 files).

Alcoholics Anonymous
Mensajes: 10
Registrado: 14 Nov 2016, 04:53

Re: RFC: Application Package Format

Mensaje por Alcoholics Anonymous » 10 Ago 2018, 07:20

aowen escribió: Ah, you're reinventing SZX only in an incompatible format that won't work with emulation. OK.
No there is a difference between a real machine format and an emulator format. This is a real machine format and it will work on any emulator that emulates the machine. An emulator format is implemented by the emulator outside of the machine emulation and requires special code to be written by the emulator author to implement it.

The problem with szx is that it is a snapshot format and contains a lot of extraneous blocks defining device state, many of which would not be definable on real hardware in an szx-snapshot loader. It's defined for big machines and would require a lot of z80 code to fully implement. It's also not able to define interactions with the operating system as the program is loaded. For example, I have to request memory pages from the system before loading into them. This is not possible to do in a snapshot format.
One is the big sna which is a standard 128k snapshot with data for more memory banks appended.
SNA is more or less the Mirage Microdriver format. 128K SNA snapshots are Pentagon snapshots. esxDOS may treat them as 128 snapshots, but that's not what they are.
Although some emulators force pentagon mode for 128k snas, 128k snas were common at one point for representing snapshots of 128k games.

sna snapshots are not proper formats for programs and the extensions only exist because of the limitations of some assemblers which can only output 128k snas as a standard form and of some emulators that can only take snas as input. As mentioned, one of the most important zx next emulators can only take sna as input (and now nex) so it's natural to extend the format to make it easy to load up all of the memory space in one file instead of having a bunch of separate files that have to be loaded by the program. Keep in mind there are up to 112 16k banks on the zx next. Ensuring that these also work on the real machine means emulation and real hw testing can happen together.
Jim's big file format is called NEX
How many ... I think I'll go put the kettle on.
There are two different requirements going on. One is for loading a program that takes over the machine and the other is for loading big programs that cooperate with the operating system. Nex is for taking over the machine and is a simple format easily generated by common assemblers. However, cooperation is where most things will be going. But you have to convince people that the machine is no longer a 48k/128k spectrum where you unplugged the machine after each game loaded. The spectrum community commonly uses very simple tools and making programs that cooperate with the system requires manual work on these simple tools.
So it's .snapload with an environment PATH variable. I'll take a look at it and see if I can support it on UnoDOS 3 without breaking compatibility. An esxDOS version running on the ZX-Uno should be able to support .tap, .sna, .z80, .p and .o (assuming I remember correctly and those last two are ZX81 and ZX80 files).
Yes .p and .o are zx81 / zx80. The program will work fine on standard esxdos but the next-specific code has to be stripped out first and the NextOS snapshot loader has to be replaced with .snapload. The code for snapload will be very similar to the nexload code in there already. I will get around to doing this but only after I've gone through the dot command generation one more time.

Avatar de Usuario
aowen
Mensajes: 178
Registrado: 07 Oct 2015, 13:32

Re: RFC: Application Package Format

Mensaje por aowen » 14 Ago 2018, 11:43

Alcoholics Anonymous escribió:
aowen escribió: Ah, you're reinventing SZX only in an incompatible format that won't work with emulation. OK.
No there is a difference between a real machine format and an emulator format.
SNA originated on real hardware (the Mirage Microdriver) and was the dominant snapshot format for emulators until Z80 came along.
An emulator format is implemented by the emulator outside of the machine emulation and requires special code to be written by the emulator author to implement it.
It's just data in a wrapper, like a RIFF file.
The problem with szx is that it is a snapshot format and contains a lot of extraneous blocks defining device state, many of which would not be definable on real hardware in an szx-snapshot loader.
That isn't a problem at all. You can skip the blocks you're not interested in, and you can write sensible defaults to enable the snapshot to work with emulators.
It's defined for big machines and would require a lot of z80 code to fully implement.
No it wouldn't. It's no harder to support than SNA or Z80. The only tricky thing would be supporting compressed SZX, but I've always said uncompressed SZX would be the way to go.
It's also not able to define interactions with the operating system as the program is loaded. For example, I have to request memory pages from the system before loading into them. This is not possible to do in a snapshot format.
It's entirely possible to do. You can read the memory page from the header, set the page, and then read the data into memory.
Although some emulators force pentagon mode for 128k snas, 128k snas were common at one point for representing snapshots of 128k games.
Not really. Z80 was far more common.
sna snapshots are not proper formats for programs
I agree, but SZX is a perfectly good format for programs. It's a shame some more recent emulators are still using inferior snapshot formats.
Keep in mind there are up to 112 16k banks on the zx next.
Terrible design decision. The extra memory on the 128 is barely useable as it is due to the awful paging system. Why wouldn't you just use the SD card?
There are two different requirements going on. One is for loading a program that takes over the machine
SZX is perfect for that.
The other is for loading big programs that cooperate with the operating system.
An auto-running BASIC program loaded from SD card should do for that. I don't see the need to dump it all in a single file when we have folders.
But you have to convince people that the machine is no longer a 48k/128k spectrum
Based on the case, that's what most people think it is. Under the hood it's a heavily modified +3, the least popular model.
The spectrum community commonly uses very simple tools and making programs that cooperate with the system requires manual work on these simple tools.
Coding for the 128 was already a drag. I can't understand why anyone would want to deal with the hassle of developing for the Next, especially with such a limited audience. But so long as it makes people happy I guess.
Yes .p and .o are zx81 / zx80. The program will work fine on standard esxdos but the next-specific code has to be stripped out first and the NextOS snapshot loader has to be replaced with .snapload. The code for snapload will be very similar to the nexload code in there already. I will get around to doing this but only after I've gone through the dot command generation one more time.
Cheers!

Alcoholics Anonymous
Mensajes: 10
Registrado: 14 Nov 2016, 04:53

Re: RFC: Application Package Format

Mensaje por Alcoholics Anonymous » 18 Ago 2018, 07:55

The problem with szx is that it is a snapshot format and contains a lot of extraneous blocks defining device state, many of which would not be definable on real hardware in an szx-snapshot loader.
That isn't a problem at all. You can skip the blocks you're not interested in, and you can write sensible defaults to enable the snapshot to work with emulators.
These blocks also apply to the machine and the point is this is sometimes hidden state that cannot be set on the machine itself even though you may want to. You can ignore them but then what you're doing is bending a snapshot format to suit another purpose.
It's also not able to define interactions with the operating system as the program is loaded. For example, I have to request memory pages from the system before loading into them. This is not possible to do in a snapshot format.
It's entirely possible to do. You can read the memory page from the header, set the page, and then read the data into memory.
You don't know what memory pages are available - you have to ask the running operating system what pages you can load into. This is not something you can statically store in a snapshot file.

Putting this into a snapshot format defeats the purpose of a snapshot format and makes it unsuitable for an emulator. Emulators need to know exactly what the state of the machine is to resume execution - they don't want to run code on the target machine to find out how to load a snapshot.

By definition the snapshot format is defining the state of the entire machine whereas a machine format needs to load on a machine that might have other things already in memory.
Not really. Z80 was far more common.
... for emulators :) snas were always preferred for real hw because it takes very little effort to load them. You'll find the archives dominated by z80 but then the people running the snapshots on their computers would use snapconv to convert to sna.
Keep in mind there are up to 112 16k banks on the zx next.
Terrible design decision. The extra memory on the 128 is barely useable as it is due to the awful paging system. Why wouldn't you just use the SD card?
The banking system on the next is very good. The standard 16k banking is retained for legacy compatibility and it has been extended with the addition of port 0xdffd so you can reach 112 16k banks using the legacy banking. This port is taken from a Russian spectrum model for compatibility there.

However the native next banking divides the same memory into 224 8k pages and divides the z80's memory space into eight 8k mmu slots. So any mmu slot can contain any 8k page.
The other is for loading big programs that cooperate with the operating system.
An auto-running BASIC program loaded from SD card should do for that. I don't see the need to dump it all in a single file when we have folders.
It makes a mess if you have a few dozen files in their own dir for each program. It's much nicer and easier to distribute if there is one file not to mention much easier for quick testing while developing. You can distribute a folder but I wouldn't do that if there is a choice. The difference becomes more significant when you get into programs that are megabytes in size.
Coding for the 128 was already a drag. I can't understand why anyone would want to deal with the hassle of developing for the Next, especially with such a limited audience. But so long as it makes people happy I guess.
There is no such thing as doing anything for the spectrum except for fun. You can't get much more than beer money on this. However, I would say you're more likely to reach more people programming the next than programming the standard spectrums - the active user base on the next is probably larger than for the standard zx, mainly because people have paid money for something recently and that demonstrates an interest and commitment for something that's more than a standard zx.

If you've got something good, I don't think you'd reach many more than 100 people on a spectrum release after all. Even if modern implementations of the standard zx are sold in the hundreds, the people buying them are most interested in playing old games rather than new.

Avatar de Usuario
aowen
Mensajes: 178
Registrado: 07 Oct 2015, 13:32

Re: RFC: Application Package Format

Mensaje por aowen » 22 Ago 2018, 11:56

Alcoholics Anonymous escribió:There is no such thing as doing anything for the spectrum except for fun.
The amount of money raised by the Elite, RCL, and Olifiers crowd-funding campaigns suggests otherwise.
I would say you're more likely to reach more people programming the next than programming the standard spectrums - the active user base on the next is probably larger than for the standard zx, mainly because people have paid money for something recently and that demonstrates an interest and commitment for something that's more than a standard zx.

If you've got something good, I don't think you'd reach many more than 100 people on a spectrum release after all. Even if modern implementations of the standard zx are sold in the hundreds, the people buying them are most interested in playing old games rather than new.
I seriously doubt that. There are thousands of emulator users who can run programs written for the 128 and there are plenty of retro sites that promote new releases. There are almost certainly more devices out there running esxDOS than the total number of Nexts that will ever be produced, but to date I can think of one game that actually uses esxDOS.

Avatar de Usuario
aowen
Mensajes: 178
Registrado: 07 Oct 2015, 13:32

Re: RFC: Application Package Format

Mensaje por aowen » 23 May 2019, 14:06

I've created an application package format for SE Basic IV 4.2. I'm considering creating a dot command version that would enable programs that don't depend on the ROM to run on esxDOS as well. Applications are stored as a collection of files and folders within a folder in a top-level folder called PROGRAMS. Application names can be any length and contain any valid FAT32 filename character (including space). In SE Basic, applications are launched with RUN "appname". The dot command equivalent would be .run appname. The application name is truncated to derive the folder and binary filenames and spaces are converted to underscores. The package format is as follows:

Código: Seleccionar todo

PROGRAMS
|----------- MY_APP_N.AME
             |----------- PRG
                          |----------- MY_APP_N.PRG
                          RSC
                          |----------- RESOURCE.BIN
The RUN command sets the SP to $6000, loads the binary (PRG file) at $6000, changes the path to the resource (RSC) folder and calls $6000. If the binary fails to load it restores the stack and falls back to BASIC. This means the binary can be up to 40K in length. The binary is then responsible for loading its own resources. The method for passing parameters to the app is to define a variable in BASIC.

This approach has a number of advantages over the single executable file method. For example, with multi-lingual software, only the selected language resources need to be loaded. It is also easy to customize the app without the need to recompile it.

Responder