Saturday, April 8, 2017

Nesting of narratives and text processsing

The puzzle is how to fit a complex narrative to a text, where the complex narrative nests sub narratives. I had the following thought while riding the Red Line somewhere around Central Square: 

 - First text is processed by being broken up into tokens. The result is called "tokenized text."
 - Next tokens are processed by conversion into a sequence of VARs, including NULL_VAR for tokens that are not recognized. The result is called "segmented text".
 - Now the smallest pieces of sub narrative can be "rolled" over the segmented text to produce a processing of the text into higher and higher order results - you could call them "n-segmented text"....and so on with n increasing with the order of the narrative.

No comments:

Post a Comment