SGML Parsing.

Anon

I have about 130 meg of text files that need to be parsed. It's a dictionary and it's all in the form of sgml. (at least that's what I assume, they're all named unab-a.sgm).

Each entry of the dictionary is on ONE line. So we have massively huge lines...

I do not know how to approach this. I tried throwing it into a dom xml parser... (xmldoc($data), where $data is one of the huge lines, always throws errors all over the place. It seems to be upset over stuff like <Key ID=5> it'd like it to be <Key ID='5'>... but well... I'm not about to go and edit all these files.)

Please help... maybe I'll just get java working on this machine. Java is supposed to have a good dom xml parser, isn't it?
Ryan

Anon

You can't normally use an XML parser for SGML documents.

If you have 130MB of data I'd suggest that you don't use DOM-style parsers unless you have a LOT of memory. There are a lot of SGML parsers around. Try James Clark's SP. I haven't tried it myself yet (but I will soon), but as far as I know it fully conforms to the latest SGML annices.

Alternatively, you can use an SGML-to-XML converter. IIRC, James Clark made one (www.jclark.com).

An XML parser is required to stop and flag an error if the parsed document isn't 100% valid XML. In XML, all attribute values MUST be literals, that is, attribute="value". Also, the markup most likely contains unclosed elements, which is forbidden in XML.

Good luck!

Anon

Thanks for replying, but I already have all the data parsed now.

I found those parsers you referred to and tried them... it turns out that not only were all the files sgml, but they were really badly formed sgml.

The parsers wouldn't work, and the converter to xml wouldn't work.

So... I read it in character by character, and created a "state" like parser.

Oy... it was a long week.