Segment Fields/Terms Iterators

[prateek]

Here's the current API spec for two methods on the Segment type:

type Segment interface {
  Fields() ([][]byte, error)
  Terms([]byte) ([][]byte,error)
} 

The return types as [][]byte have a number of detriments:

• Can't pool the underlying []byte (no easy way for user to finalise and return)
• Have to alloc the entire [][]byte upfront
• Can't wrap underlying iterators. e.g. we could wrap vellum's FST iterator if returned an iterator ourselves

Other factors to consider:

• We could enforce lexicographical ordering on the returned iterators, which would be useful when constructing FSTs, and the iterators returned by FST would natively satisfy this.
• Could tie-lifecycle of the iterator being returned to the segment's lifecycle - e.g. FST iterator wrapper would only be valid for as long as the iterator itself.