Syntactic ambiguity resolution in the GHC parser

There are places in the Haskell grammar where it's not known apriori whether it's an expression a command or a pattern that is being parsed. This used to be handled by picking a parse (e.g. as an expression say) and if that choice later turned out to be wrong, "rejigging it" (transform the constructed parse tree to its analog in the pattern language). The problem with that approach is that it meant having conflated sub-languages meaning, for example, HsExpr had to have pattern related constructors e.g. EWildPat, EAsPat (and further, these propogated into other compiler phases like the renamer and typechecker). This was the case until roughly a year ago before extraordinary work by Vladislav Zavialov who solved the ambiguity resolution issue by parsing into an abstraction with an overloaded representation:

class DisambECP b where ...
newtype ECP = ECP { unECP :: forall b. DisambECP b => PV (Located b) }

This innovation might be considered to have come at a cost for developers familiar with the "old" parser however. That is, dealing with understanding the apparent complexity introduced by the ambiguity resolution system. This post attempts to provide some intuition about how the system works and hopefully will lead to the realization that it's not that hard to understand after all!

Because this post is about building intuition, there are details that are glossed over or omitted entirely: the reader is encouraged to read Vlad's detailed explanatory comments in RdrHsSyn.hs when neccessary to address that.

We start with something familiar - the GHC parser monad:

newtype P a = P { unP :: PState -> ParseResult a }

This fundamentally is a wrapper over a function PState -> ParseResult a.

The (let's call it the) "ECP system" introduces a new (and as we'll see, very related) concept. The parser validator monad:

newtype PV a = PV { unPV :: PV_Context -> PV_Accum -> PV_Result a }

So a parser validator is a function similar in spirit to a parser where:

• data PV_Context: The type of essentially a wrapper around the lexer ParserFlags value;
• data PV_Accum: The type of state accumulated during parsing validation (like errors & warnings , comments, annotations);
• data PV_Result: The parser validator function's result type that is, data PV_Result a = PV_Ok PV_Accum a | PV_Failed PV_Accum.

Of critical interest is how this type is made a monad.

instance Functor PV where
  fmap = liftM

instance Applicative PV where
  pure a = a `seq` PV (\_ acc -> PV_Ok acc a)
  (<*>) = ap

The above reveals that an expression like return e where e is of type Located b, constructs a function that given arguments ctx and acc returns e. The moral equivalent of const.

instance Monad PV where
  m >>= f = PV $ \ctx acc ->
    case unPV m ctx acc of
      PV_Ok acc' a -> unPV (f a) ctx acc'
      PV_Failed acc' -> PV_Failed acc'

The bind operation composes PV actions threading context and accumlators through the application of their contained functions: given an m :: PV a and a function f :: a -> PV b, then m >>= f constructs a PV b that wraps a function that composes f with the function in m.

PV is a bit more than a monad, it also satisfies the MonadP class for monads that support parsing-related operations providing the ability to query for active language extensions, store warnings, errors, comments and annotations.

instance MonadP PV where
  addError srcspan msg = ....
    PV $ \ctx acc@PV_Accum{pv_messages=m} ->
      let msg' = msg $$ pv_hint ctx in
      PV_Ok acc{pv_messages=appendError srcspan msg' m} ()
  addWarning option srcspan warning = ...
  addFatalError srcspan msg =...
  getBit ext =
    PV $ \ctx acc ->
      let b = ext `xtest` pExtsBitmap (pv_options ctx) in
      PV_Ok acc $! b
  addAnnotation (RealSrcSpan l _) a (RealSrcSpan v _) = ...
  ...

The function runPV is the interpreter of a PV a. To run a PV a through this function is to produce a P a.

runPV :: PV a -> P a

That is, given a PV a construct a function PState -> ParseResult a.

runPV m =
  P $ \s ->
    let
      pv_ctx = PV_Context {...} -- init context from parse state 's'
      pv_acc = PV_Accum {...} -- init local state from parse state 's'
      -- Define a function that builds a parse state from local state
      mkPState acc' =
        s { messages = pv_messages acc'
          , annotations = pv_annotations acc'
          , comment_q = pv_comment_q acc'
          , annotations_comments = pv_annotations_comments acc' }
    in
      -- Invoke the function in m with context and state, harvest its revised state and
      -- turn its outcome into a ParseResult.
      case unPV m pv_ctx pv_acc of
        PV_Ok acc' a -> POk (mkPState acc') a
        PV_Failed acc' -> PFailed (mkPState acc')

It is often the case that a production (or set of productions) might result different ASTs depending on the context. Ideally, we just want to write the productions once and reuse them across these different sub-languages (e.g. expressions vs. commands vs. patterns). For example, the production for a parenthesized "thing" is

'(' texp ')'

In the context of a pattern we expect an AST with a ParPat _ p node whereas in the context of an expression we want an AST with an HsPar _ e node. To this end the DisambECP class embodies an abstract set of operations for parse tree construction.

class DisambECP b where
  ...

  -- | Return a command without ambiguity, or fail in a non-command context.
  ecpFromCmd' :: LHsCmd GhcPs -> PV (Located b)
  -- | Return an expression without ambiguity, or fail in a non-expression context.
  ecpFromExp' :: LHsExpr GhcPs -> PV (Located b)

  ... Lots of operations like this
  mkHsOpAppPV :: SrcSpan -> Located b -> Located (InfixOp b) -> Located b -> PV (Located b)
  mkHsVarPV :: Located RdrName -> PV (Located b)

  ...

The idea is that in the semantic actions of the grammar we construct and compose parser validators in terms of these abstract functions. Running the PVs produces parsers and at the point of execution of parsers we know the context (the nature of the AST we expect to recive) and the concrete choices for each of the abstract functions is thereby fixed (and then, on evaluation, we get the parse result).

The only wrinkle is in the return type of productions that produce parser validators. In general, they will have the form forall b. DisambECP b => PV (Located b). If they were monadic productions though we would be led to P (forall b. DisambECP b => PV (Located b) and that dog don't hunt for GHC's lack of support for impredicative types. There is a standard work-around that can be employed though. This newtype is how impredicative types in monadic productions are avoided:

newtype ECP = ECP { unECP :: forall b. DisambECP b => PV (Located b) }

So here, ECP is a wrapper around a PV (Located b) value where b can be of any type that satisifies the constraints of class DisamECP. So, in a production that looks like

| ... {% return (ECP ...)}

we are dealing with P ECP whereas without a newtype we would be dealing with P (forall b. DisambECP b => PV (Located b)).

Now to produce a P (Located b) from the PV (Located b) in an ECP we can use this expression (of type DisambECP b => ECP -> P (Located b)):

runPV (unECP p)

It takes an ECP value, projects out the parser validator contained therein and "runs" it to produce a function from PState -> ParseResult a (a parser action).

From the DisabmECP instance for HsExpr GhcPs, here's ecpFromCmd':

  ecpFromCmd' (L l c) = do
    addError l $ vcat
      [ text "Arrow command found where an expression was expected:",
        nest 2 (ppr c) ]
    return (L l hsHoleExpr)

Makes perfect sense - you get a parser validator that when evaluated will store a (non-fatal) error and returns an expression "hole" (unbound variable called _) so that parsing can continue.

Continuing, the definition of ecpFromExp':

  ecpFromExp' = return

Also sensible. Simply calculate a function that returns its provided acc argument together with the given constant expression under a PV_Ok result (see the definition of pure in the Appliciatve instance for PV given above).

Parenthesizing an expression for this DisambECP instance means wrapping a HsPar around the given e:

  mkHsParPV l e = return $ L l (HsPar noExtField e)

And so on. You get the idea.

So how does this all fit together? Consider agin the production of parenthesized things:

        | '(' texp ')'  { ECP $
                            unECP $2 >>= \ $2 ->
                            amms (mkHsParPV (comb2 $1 $>) $2) [mop $1,mcp $3] }

We note that the texp production calculates an ECP. Stripping away for simplicity the annotation and source code location calculations in the semantic action, in essence we are left with this.

ECP $ unECP $2 >>= \ $2 -> mkHsParPV $2

The effect of unECP is to project out the forall b. DisambECP b => PV (Located b) value from the result of texp. Recalling that unPV projects out the function that the PV wrapper shields and by substition of the definition of bind, we obtain roughly:

  ECP $ PV $ \ctx acc ->
                case unPV (unECP $2) ctx acc of
                  PV_Ok acc' a -> unPV (mkHsParPV a) ctx acc'
                  PV_Failed acc' -> PV_Failed acc'

The net effet is we construct a new parser validatior (function) from the parser validator (function) returned from the texp production that puts parenthesis around whatever that function when evaluated produces. If used in a context where texp generates a LPat GhcPs that'll be a ParPat node, if an LHsExpr GhcPs, then a HsPar node.