Previous Thread
Next Thread
Print Thread
Page 2 of 7 1 2 3 4 5 6 7
Re: Sub-channel support for CDs [Re: Fake Shemp] #106273 07/02/16 07:30 AM
Joined: Dec 2013
Posts: 135
X
xinyingho Offline
Senior Member
Offline
Senior Member
X
Joined: Dec 2013
Posts: 135
Now I understand better why having data in scrambled form is better. I didn't think about those CDs with illegal sector length.

But then, I agree with R. Belmont. For CDs, the ideal form should be the original "analog" bitstream of pits and lands.

It's funny that Haze talks about the error correction data being generated on the fly because it got me to look again at the algorithm of how CDs are translated from channel frames (the pits and lands, or the raw level) to sections (the low level, which contains the subchannel data missing in sectors) and finally to sectors (the high level, where data don't need to be transformed again to be understandable by games / applications). Looking at this, I don't understand how you come to 2448 bytes/sector.

Let me explain by going this algorithm the other way around from sectors towards channel frames.

1/ Everybody knows by now that having an image of 2048 bytes/sector doesn't work out for a general-purpose image format because not every CD-ROM are formatted as 2048 bytes/sector. There are at least 2 other formats (2324 and 2336 bytes/sector), which eat on the Ctrl/ECC/ECD parts of sectors to lengthen the data part of 2048 bytes. But going for a "raw" image of sectors with 2352 bytes/sector is better because then we can account for any possible sector formats including CD-DA (i.e. CD audio).

2/ To translate sectors to sections, sectors' raw data are scrambled and split into F1-frames of 24 bytes. To a F1-frames, we add 8 bytes of error correction data, giving a F2-frame. Then we add 1 byte of subchannel (containing the TOC and other useful information spread out into an entire section), giving a F3-frame of 33 bytes. A section is made up of 98 F3-frames.

I don't need to go to the 3rd and last step of translating sections into channel frames as no devices allow us to get data in that last form.

So at the low level, data are organised as sections of 3234 bytes (33 bytes * 98). If we remove the useless error correction data that can always be recomputed on the fly, I come to sections of 2450 bytes (3234 bytes - (8 bytes * 98)), not 2448 bytes/section. What am I missing?

Last edited by xinyingho; 07/02/16 07:36 AM.
Re: Sub-channel support for CDs [Re: xinyingho] #106279 07/02/16 03:47 PM
Joined: Oct 2007
Posts: 300
F
F1ReB4LL Offline
Senior Member
Offline
Senior Member
F
Joined: Oct 2007
Posts: 300
Originally Posted By xinyingho
1/ Everybody knows by now that having an image of 2048 bytes/sector doesn't work out for a general-purpose image format because not every CD-ROM are formatted as 2048 bytes/sector. There are at least 2 other formats (2324 and 2336 bytes/sector)

And CDDA uses 2352/sector. And CD+G uses all 2448/sector. And there are games with CD+G tracks, like the Japanese Mega CD Wondermega disc.

Originally Posted By xinyingho
Now I understand better why having data in scrambled form is better. I didn't think about those CDs with illegal sector length.


It was only a brief example about why is it inaccurate to use the descrambled format to store the images, there are dozens of other cases. For example, there's a common mastering error when there's an empty sector (2352 bytes of zeroes) in the data track - how are you going to "descramble" it? Or it can be a half-data, half-empty sector - while it easily passes the descrambler if the first part of the sector is filled with data and the subheader is there, what are you going to do with the sector, where only the 2nd half has data? Or with various non-standard sectors (errorneous or protection-related) filled with data but have no subheaders? And about the current way of storing the CDs without the first 150 pregap sectors: how are you going to handle many of the Saturn CDs, where 2 of the first pregap sectors are doubled resulting the real pregap size is 152 sectors? You can't save this information in the cuesheet. And if you cut only the first 150 sectors leaving everything else as is, the image becomes unusable (at least, with the current way of handling the CD images). All the known images have the first 152 sectors cut, but you can't call these images "precise". That's why you can't have even a barely precise emulation/preservation using the descrambled images without the first pregap/lead-in/lead-out.

Re: Sub-channel support for CDs [Re: F1ReB4LL] #106281 07/02/16 07:05 PM
Joined: Dec 2013
Posts: 135
X
xinyingho Offline
Senior Member
Offline
Senior Member
X
Joined: Dec 2013
Posts: 135
Originally Posted By F1ReB4LL

It was only a brief example about why is it inaccurate to use the descrambled format to store the images, there are dozens of other cases. [...]

I see. You've convinced me that unscrambled data is better to handle every cases.

Originally Posted By F1ReB4LL

And CDDA uses 2352/sector. And CD+G uses all 2448/sector. And there are games with CD+G tracks, like the Japanese Mega CD Wondermega disc.

I see. Now I understand what you mean by 2448/sector.

If we follow the terminology of the official ISO/IEC 10149 and ECMA-130 specifications (both overriding the Red and Yellow Book specs), CD+G still uses the standard 2352/sector format, it's just that it also exploits the normally unused part of the subchannels (by the way, the specs never use the term "subchannels" but call them "control bytes" instead). The 2 subchannels P and Q are already reserved by the specifications, so CD+G uses the remaining 6 subchannels (R to W) to store additional data. But it seems like you still include the entire subchannels (P to W) when measuring the length of a sector.

So if I take the explanations from my previous post, I should still have 2450 bytes/sections for CD+G, not 2448.

I re-read the specs and I found out that I overlooked a fact: the 2 first subchannel bytes of a F3-frame always have the same values and are used for synchronisation purposes when converting F3-frames to channel frames. So the final calculation isn't [3234 bytes - (8 bytes * 98)] but should be [3234 bytes - (8 bytes * 98 - 2 sync bytes) = 2448 bytes/section].

Finally, I got what you call 2448/sector.

Re: Sub-channel support for CDs [Re: xinyingho] #106287 07/03/16 12:59 PM
Joined: Oct 2007
Posts: 300
F
F1ReB4LL Offline
Senior Member
Offline
Senior Member
F
Joined: Oct 2007
Posts: 300
Originally Posted By xinyingho
I see. You've convinced me that unscrambled data is better to handle every cases.

Scrambled, you mean? smile

Re: Sub-channel support for CDs [Re: F1ReB4LL] #106289 07/03/16 01:07 PM
Joined: Jun 2013
Posts: 35
O
Octocontrabass Offline
Member
Offline
Member
O
Joined: Jun 2013
Posts: 35
Originally Posted By F1ReB4LL
For example, there's a common mastering error when there's an empty sector (2352 bytes of zeroes) in the data track - how are you going to "descramble" it? Or it can be a half-data, half-empty sector - while it easily passes the descrambler if the first part of the sector is filled with data and the subheader is there, what are you going to do with the sector, where only the 2nd half has data? Or with various non-standard sectors (errorneous or protection-related) filled with data but have no subheaders?

The (de)scrambler for data sectors is a simple XOR with a 2352-byte pattern. It's the same for every sector and it's independent of the sector's content. It doesn't make any difference to these examples whether the CHD internally stores them scrambled or unscrambled; the only difference will be how well the data compresses.

Originally Posted By R. Belmont
PS: taking a cue from the floppy subsystem, a bitstream of pits and lands would be the ultimate ideal form. It could represent PC copy protections that violate all possible axes of the ISO standards, for instance.

As long as the CHD decompresses to the correct bitstream of pits and lands, does it matter how the CHD internally stores the data? There are a lot of CDs that could be stored as (mostly) unscrambled 2048-byte sectors and a handful of parameters, but decompress to the original bitstream.

Re: Sub-channel support for CDs [Re: F1ReB4LL] #106290 07/03/16 01:39 PM
Joined: Dec 2013
Posts: 135
X
xinyingho Offline
Senior Member
Offline
Senior Member
X
Joined: Dec 2013
Posts: 135
Originally Posted By F1ReB4LL
Originally Posted By xinyingho
I see. You've convinced me that unscrambled data is better to handle every cases.

Scrambled, you mean? smile

LOL yes, I meant scrambled.
But then again Octocontrabass seems to have an argument saying that being scrambled or unscrambled doesn't actually matter. If you really can convert data forth and back without data loss in all situations, then effectively it shouldn't matter. But I'm all for demonstrations with actual data to be convinced. After all, we're talking about technical subjects based on mathematical logic. There shouldn't be any doubts about facts deriving from mathematical principles.

Originally Posted By F1ReB4LL

Reason 1: Because that's the way it's stored on real CDs (MAME doesn't use decrypted NeoGeo ROMs, because they are encrypted inside the chips, why should the CDs be stored in descrambled form?)

I just wanted to correct you on this assertion. Data on real CDs aren't stored as scrambled data, this form only exists as intermediate working state in driver chips when converting channel frames to sectors.
Even 2448/sector doesn't exist at all. Following my previous messages and the demonstration they give, you arrive at this number by removing the 2 first sync bytes of sections and all of the error correction data.
But I suppose that this scrambled 2448/sector format exists because this is the most complete format that people managed to dump using a particular CD driver model, which has been withdrawn from sale since.

Re: Sub-channel support for CDs [Re: Fake Shemp] #106292 07/03/16 04:06 PM
Joined: Jun 2011
Posts: 24
Z
zyrobs Offline
Member
Offline
Member
Z
Joined: Jun 2011
Posts: 24
Unlike main channels, subchannel cannot be ripped properly due to lack of error correction on the format (two identical discs will give you different ripped subs).

So it would make more sense to store them in a separate file.
Because if you combine them with main channel in a 2448 byte format (technically it would be 2450 byte since you have a 0x00 denoting the beginning and end of each sector), then you cannot get proper CRCs for any disc image. Anyone ripping their own disc will always get different CRCs, unlike with ROM dumps.

If we store main channel and sub channel in different files, this is not a problem. People can rip their discs to a matching CRC and know if they got a different revision or an existing one.

Scrambling is not necessary, as all hardware reads the disc unscrambled. It is not any more necessary as storing ROM dumps as transistor bitmaps and sound dumps in vector graphic representation of waveforms.

Re: Sub-channel support for CDs [Re: xinyingho] #106293 07/03/16 04:07 PM
Joined: Jun 2001
Posts: 422
O
Olivier Galibert Offline
Senior Member
Offline
Senior Member
O
Joined: Jun 2001
Posts: 422
I've been told to point you guys to:
https://en.wikipedia.org/wiki/Compact_Disc_Digital_Audio

which says the frames are 588 bits.

Incidentally, the guy who told me tells me he tried to follow the registration method and in addition to contact Richard and failed. Is it actually possible to register here by now?

OG.

Re: Sub-channel support for CDs [Re: zyrobs] #106294 07/03/16 04:29 PM
Joined: Oct 2007
Posts: 300
F
F1ReB4LL Offline
Senior Member
Offline
Senior Member
F
Joined: Oct 2007
Posts: 300
Originally Posted By Octocontrabass
The (de)scrambler for data sectors is a simple XOR with a 2352-byte pattern. It's the same for every sector and it's independent of the sector's content. It doesn't make any difference to these examples whether the CHD internally stores them scrambled or unscrambled

Wrong. (De)scrambling requires the proper sync in the beginning of the sector, otherwise it reports a read error and doesn't output anything.

Originally Posted By zyrobs
Unlike main channels, subchannel cannot be ripped properly due to lack of error correction on the format (two identical discs will give you different ripped subs).

There are 2 approaches to this problem: 1) Automatic correction of simple 1-bit errors (99.9% of the subchannel dumps become identical on any drive); 2) Careful analyzing of errors in the dumps taken from different copies of the CDs with the same matrix on different drives. Simple tru*cough*rip subs would be totally useless here, also, the question atm is about the format and further emulation logic, not about what dumps to put inside.

Re: Sub-channel support for CDs [Re: Olivier Galibert] #106296 07/03/16 05:05 PM
Joined: Dec 2013
Posts: 135
X
xinyingho Offline
Senior Member
Offline
Senior Member
X
Joined: Dec 2013
Posts: 135
zyrobs thanks for confirming that technically it should be 2450 bytes and not 2448.

Originally Posted By Olivier Galibert

I've been told to point you guys to:
https://en.wikipedia.org/wiki/Compact_Disc_Digital_Audio

which says the frames are 588 bits.

Saying that a frame has a size of 588 bits is irrelevant.
To explain this and why the wiki page talks about frames of 588 bits, I'll need to go to the 3rd and last step of the algorithm used to translate sectors to channel frames. So here we go:

3/ At the end of step 2, the algorithm outputs sections made up of 98 F3-frames, each F3-frame being 33 bytes in size.
From there, each F3-frame is converted to a channel frame using the EFM algorithm (Eight to Fourteen Modulation), which convert each byte of a F3-frame (i.e. 8 bits) to an equivalent 14-bit codeword. After the conversion, each channel frame is composed of a sync header of 24 bits, a control codeword (so 14 bits, in the specs "control bytes" design the subchannel, so the control codeword is the subchannel converted through EFM), and an interleaved sequence of 34 "merging channel" (3 bits each) and 32 codewords (so 14 bits each, those are the remaining F3-frame bytes converted through EFM).

So in the end, at the raw level, a channel frame is 24 bits + 14 bits + 34 * 3 bits + 32 * 14 bits = 588 bits.
But no consumer device allows us to get data in this raw format. So there's no point discussing this further...

Originally Posted By F1ReB4LL

Wrong. (De)scrambling requires the proper sync in the beginning of the sector, otherwise it reports a read error and doesn't output anything.

Ah, there you're talking about the 2 missing sync bytes of your 2448/sector. Yes, it should be 2450 bytes/section.

Page 2 of 7 1 2 3 4 5 6 7

Who's Online Now
2 registered members (RColtrane, Golden Child), 48 guests, and 2 spiders.
Key: Admin, Global Mod, Mod
ShoutChat Box
Comment Guidelines: Do post respectful and insightful comments. Don't flame, hate, spam.
Forum Statistics
Forums9
Topics8,821
Posts116,119
Members4,921
Most Online890
Jan 17th, 2020
Powered by UBB.threads™ PHP Forum Software 7.7.3