Home Page Forums MAME MAME Imgtool

(continuing this discussion on a separate topic)

I see forks being supported simply by adding a 'const char *fork' parameter to all of the relevant calls (read_file, write_file), as well as a list_forks function to list all forks of any given file.

As for things like MacBinary, there is a concept of "filters" that I added a while back but never took to its greatest potential. Essentially the idea is a filter can do some transformation on the file when the file is inserted to or extracted from an image. Currently, the only filters change endline characters and [de]tokenize BASIC programs. In theory, MacBinary support could be done at this level.

Partitioning: I'm not sure that I like the idea of having partitions behave as special directories. A consequence of that approach is that the special directory would end up behaving "weirdly" (e.g. - it would not be possible to create directories). I see directories as being implemented by adding an 'int partition' parameter to the relevant functions, and also adding a handful of functions to implement partition maps.

As for TI key indexed files, can you explain them to me (how they work and what they are used for?) It would be easier to determine what core changes are necessary if I understood them better.

I can go ahead and add the relevant parameters to those functions for partitions and forks, even if we do not fully support them in the core yet. That would allow the Mac Imgtool driver to be further developed along these lines, while I get my act together in the core. Comments?

Forks:

The idea of passing fork names makes sense to me, as HFS+ and (IIRC) NTFS identify file forks with their names, and emulating fork names under MFS/HFS should be straightforward.

Should we define the fork name character encoding to be in Unicode?

I think there might be a special value for the data fork, as it not entirely unlikely that the data fork should have different filesystem-specific names accross different filesystems.

One problem I can see, however, is that we cannot have NUL characters in C strings. It is currently a problem with the macintosh imgtool driver, as MFS and HFS do not prohibit file names from including NUL characters (though it is not recommended); it will be a problem if we ever implement HFS+ where fork names seem to follow the same rules. As an interim solution, the macintosh imgtool driver escapes NUL characters; however, this solution is not particularly elegant, as it is obvious that imgtool drivers are not the place to implement character escaping: IMO, it should be implemented in the osd interfaces, as each OS may have different character set requirements, though the imgtool core may provide utilities and frameworks to help osd code to do character escaping.

Also, if we support extracting specific forks, how would we support MacBinary extraction of both data and resource fork? Should we define a special value for the fork reference that would act as a wildcard and read both forks as macbinary? Such a design does not sound extremely clean to me. An alternative might be adding, at a level over the raw fork access API, another API that would support elaborate formats such as macbinary.


Macbinary filters:

Creating filters can be a solution. However, this removes nothing from my remarks about file meta-information; it would make sense for the user to view and edit individual bits of file meta-information.


Partitioning:

But if some free space remains on disk, we may create new partitions on the hard disk (provided we have added imgtool interfaces to support it and the imgtool driver supports it).

Maybe the term directory is not a good term: a better term would be entity.

When we list a partition map, we get a list of partitions; when we list a directory, we get a list of sub-directories and files: all these make an entry in the list, which I call an entity. I agree that there could (and should) be something to mark the difference of nature between entities; for an example a different icon for files, directories or partitions, and/or a text field that would explicitely say "file", "directory", "partition", etc.

Here is how I would see interaction in imgtool GUI: the user opens a hard disk image; a window opens with a representation of the partition map; each partition is a distinct entity, that appears as a line in the list. Some partition (e.g. unix swap partition, macintosh driver partition) may be data partition that cannot be listed: the user could extract the data in these partitions with the GET command, and possibly owerwrite it with PUT. Other partitions may be file system partitions that can be listed: the user could then list these partitions by opening them; or, alternatively, he could run GET to extract the partition file system as a stand-alone disk image. With CLI, the user could list the partition map by listing the partition, and he would access respective partition by prefixing the path with a partition identifier.

What do you say?


TI key indexed files:

Don't worry too much about TI key indexed files, as I don't plan to add support for them in the near future. Anyway, here is a rundown of TI formats:

* fixed-length record formats (TI 99 and 990): these formats organize data as a succession of flat records, all of the same length

* variable-length record formats (TI 99 and 990): these formats organize data as a succession of flat records of varying length; data therefore cannot be regarded as a flat byte stream, since we need to keep the lenght of each record

* key-indexed files (TI 990 only): once again, the data is organized as a succession of records; however these records are not flat, but structured as a succesion of data fields; some fields of the structure can be defined as key fields: the file system consequently maintain a tree structure for each key field, so that the file can be rapidly searched for records that have a given value in a given key field; as you can guess, this is used mainly for database-related tasks

I don't know if there is a general way to support. I thought file meta-information, including file structure information, could be stored in an XML file, but I have not had the time to elaborate the idea. Anyway, I think we can keep all this as a long-term reflexion project.


It is OK with me.

Unicode (specifically UTF-8) makes sense, because thats what I use for filenames.

Yuck! At some level, NUL characters used this way must be escaped in some way. The only alternative is to rework the entire Imgtool API so that it supports file and fork names that are not conventional C strings. IMHO that is not worth it for what amounts to such an extreme edge case. An alternative is to have the Mac driver representing NUL characters as some other Unicode code point, possibly one that is in the private use area or even one that is technically illegal (for example U+FFFFFFFF). If other drivers do that, we could standardize on a specific code point. (Does the current driver currently convert UTF-8 to and from MacRoman?)

I'd actually prefer to avoid that approach, and have all MacBinary functionality in a filter (once filter infrastructure is adequate) because it would create behavior on the part of the driver of which the front end is not directly aware. I would rather have the functionality separated out, and enhance filters to support all of the intelligence to enable front ends to allow chosing MacBinary with the same user friendliness as programs like HFVExplorer.

This is true; we also need an API for the metadata. A hypothetical MacBinary filter could then use this API to do its business.

I see what you're getting at. I'm not sure one way or another right at this point whether that could be done. Needless to say, I think that I have my work cut out in the medium term with the above issues.

Good. Incidentally, I can see that mac imgtool is really lagging behind its windows counterpart, I haven't addressed character encoding yet.

I understand that you prefer keeping C strings to re-writing all the imgtool drivers. Still, there should be standard way of escaping NUL characters, that could be used as well in the GUI/CLI to escape other control characters (or merely characters whose glyph is unavailable on the host computer, for instance few non-chinese computers have chinese ideograph fonts, and I doubt many non-apple computers have the Apple logo that is part of standard macintosh fonts).

Nope. Anyway, it is not just a matter of MacRoman, HFS supports file names in any of the macintosh character encodings, which include MacRoman, greek, cyrillic, japanese, korean, chinese, among others. There is no character encoding conversion right now; I guess I should add character encoding conversion tables at some point in the future. As if the macintosh imgtool driver were not already big enough as it is.

Understood. However, filter or no filter, we would get different objects according to which format we use: in raw format, we would get and put a single file fork; in macbinary and applesingle format, we would get and put the whole file. As to appledouble, it may be implemented as a two step process: one step would be getting/putting the resource fork in appledouble format, the other step would be getting/putting the data fork in raw format. I have no idea how to implement this under a GUI without making the average user utterly confused. Any idea?

Please, do get the time to think the things carefully. I think that my idea of listing the partition map really makes sense.

Regards,

Raphael

I just Google'd for "apple logo macroman" and found this tidbit detailing how (assuming that it is accurate) Apple encodes its logo at U+F8FF for purposes of translating MacRoman to Unicode. Similarly, bonafide NULs in filenames can be encoded as something else (maybe U+E000, U+F8FE or something else in the private use area). I don't see the exact choice as that important; we can reference it as a constant and change it later if we wish.

If you have access to a Windows box, you should take a look at HFVExplorer. In HFVExplorer, the choice of AppleDouble/MacBinary/raw is on a dropdown menu within the insert/extract dialog. Its even smart enough to make some elementary decisions (i.e. - default to raw for text files, MacBinary for applications). This would imply that filters would do more than strictly filter, but also have some logic that allows front ends to make these sort of decisions.

I will; I'd like to get these fork/metadata issues resolved before I tackle partitions. I am itching to use the Mac driver without jumping through the hoops described in the other thread.

[QUOTE]I just Google'd for "apple logo macroman" and found this tidbit detailing how (assuming that it is accurate) Apple encodes its logo at U+F8FF for purposes of translating MacRoman to Unicode. Similarly, bonafide NULs in filenames can be encoded as something else (maybe U+E000, U+F8FE or something else in the private use area). I don't see the exact choice as that important; we can reference it as a constant and change it later if we wish./QUOTE]

Yes, defining private code points to implement the NUL character and such system-specific characters as the Apple logo internally (i.e. at the imgtool core and driver level) would make sense. Still, think I have a valid point with undisplayable characters, and it would be a good idea to define a standard way, across imgtool implementations, to escape these characters on the GUI (and possibly CLI) level.

[QUOTE]If you have access to a Windows box, you should take a look at HFVExplorer. In HFVExplorer, the choice of AppleDouble/MacBinary/raw is on a dropdown menu within the insert/extract dialog. Its even smart enough to make some elementary decisions (i.e. - default to raw for text files, MacBinary for applications). This would imply that filters would do more than strictly filter, but also have some logic that allows front ends to make these sort of decisions./QUOTE]

I guess that each imgtool driver would have a list of filters that make sense with this driver: in this scheme, macintosh might propose raw, macbinary and applesingle, ti-99 might propose tifiles and fiad (two common ti-99 file encoding), coco would propose raw and BASIC token decoder, and fat would probably only propose raw.

Still, selecting raw format would not be the end of the story, as on a macintosh, you may get either the data fork or the resource fork in raw format. To implement fork selection in raw format, we might propose a filter-specific option: when the raw format is selected, the fork selection would be implemented as a pop-up menu or a combo box (or anything similar): it would probably default to data fork but would let the user select the resource fork if he wants to extract it.

Regards,

Raphael

Wouldn't it make more sense to have the mac propose rawdata and rawresource filters or some such?

Justin, your idea could make sense, too. I was thinking of forks in the general case: if we implement named forks in the most general case, then it is normal for the user to give the name of the fork to get/put. If we just care about MFS/HFS, then it is probably simpler to create a rawdata and a rawresource filter.

Regards,

Raphael

Raphael, what is the "name" (as in named fork) of the Mac resource fork? When I add named fork capability to the Imgtool core, I will use the Apple II ProDOS driver as a guinea pig, since I wrote that driver and MacBinary is also used for that system.

Since simulating resource forks "on top" of a named fork abstraction seems to be the way to go, it makes sense for ProDOS to use the same resource fork name that MacOS uses.

In MFS and HFS, the forks have no name; however, in HFS+, the respective names are "" (an empty string) for the data fork, and "RESOURCE_FORK" for the resource fork, so to keep it consistent between different Apple file systems, these are the names that we should simulate. I did not even know ProDOS supported multiple forks, BTW.

Regards,

Raphael