/ GDC, DATA, DETERMINISM, DDL

Notes for "The Future of Scene Description on 'God of War'" GDC2019

Reading time: 5 minutes

TLDW Summary: This talk sheds light on the decisions that were made by the Santa Monica Studio’s engine team while tackling the problems of time and complexity in transforming source data into game-ready data.

Keywords

  • Data Design
  • Data Duality
  • Scene Description
  • Data Definition Language (DDL)
  • Nondeterminism

Presentation Time Stamps

Backgound: Game Content Pipeline

timestamp: 1 min into the presentation

  • God of War data breakdown slide: God of War data breakdown
  • What Source Data and Run-time data have in common slide: What Source Data and Run-time data have in common
  • The design philosophy was inspired by the scene description from the film industry
  • Patch size was a common enemy
    • Nondeterminism made patch sizes huge
      • Patching was always an afterthought.
      • Parts of the run-time was an afterthought.
  • Exploring data formats to find a better way
  • 2 types of explicit data formats
    • Low level: FlatBuffers, ProtocolBuffers
    • Higher level: COLLADA and USD
  • None satisfied the requirements
    • This was the catalyst for the SmSchema Data Definition Language
  • The SmSchema proof of concept had 3 main parts
    • The definition itself
    • C++ code generation with (Jinja Python)
    • Sterilization into JSON
  • The first attempt to apply it to a real-world problem showed that the mental model that they had envisioned had missing parts slide: missing interconnected technologies

Past: Redefining SmSchema

timestamp: 16 min into the presentation

  • Decided to use the serialization as an anchor point to design around
  • The redesign started with laying down the foundation with first-prenciples slide: first-prenciples used in the design
  • Wanted to get away from a JSON-based Language
  • Issues with Maya scalability brought more questions that needed answers slide: questions that needed answers

Present: A view after years of investment

timestamp: 20 min into the presentation

  • slide: SmSchema architecture high level
  • The DDL was inspired by Insomniac’s DDL
  • Used ANTLR for language definition
  • Process of code generation
    • Used special Compile-Time Type Info (CTTI) to decouple the DDL and the code generation
    • Used a functional approach to make it easier to generate code to any current or future target
    • Used special metadata in the form of annotations to enable user-driven code generation without making changes in the parser
  • slide: Code Generation with CTTI info
  • slide: Code Generation with CTTI flexibility
  • Serialization formats
    • JSON - Human readable for merging, diffing and fixing issues
      • Leverage open source tools and libs
    • Binary - For performance & compression
    • Experimenting with MsgPack to encode large assets
    • X3D scene loader increased performance by an order of magnitude while saving and loading

Present: connecting the parts of the design into one

timestamp: 27 min into the presentation

  • The defining structure of SmSchema that brings all of the parts together is called the document.
  • The Document is a “Section of data with a header”
    • Defining the semantics of pointers used in the referencing solution. Used to reference other documents.
  • Pointers was a common request for SmSchema
    • See details on the implementation of pointer @ min 28
    • The pointer had to provide
      • Ownership semantics: unique, weak
      • Locality semantics: local, section, external
      • Used Frozen/Alive concept to determine if the reference is loaded
      • The concept of the section was introduced to SmSchema
        • A document is segmented into several memory blocks
        • These memory blocks are called sections and are used for storing things like debug data or GPU data
  • slide: Sections of the SmSchema Doc
  • With the Document structure, they were able to design workflows with a minimum amount of transformations from the source data to the run-time data
    • The document could be
      • Shipping streaming resource
      • A string hash look-up table
      • A source X3D scene graph
  • slide: Transforming data from source to run-time

Looking into the future: many opportunities still remain

timestamp: 37 min into the presentation

  • Current and future research slide: SmSchema architecture high level Final
  • Challenges with external pointer resolution
    • Who is responsible for resolving? slide: Frozen external pointers
  • This led to the creation of a data structure for arbitrating external pointer resolution called the “document store”
    • The document arbitration takes a form of transaction slide: Document store transactions
  • The combination of the Document Store and Streaming Policies could produce complex decision-making behavior for streaming resources

"Solutions have to be driven by use case, introduced with scrutiny, and solve a domain of problems evidently and clearly."

Lessons we’ve learned

timestamp: 41 min into the presentation

  • While answering what is the future of Scene Description on “God of War”
    • They understood that there are other questions that they need to answer before answering the original question
      • What are the primitives used in the design of
        • The engine
        • The workflows
        • The formats

"Our goal was never to predispose ourselves to a particulate design. The future of scene description is not a data structure, it’s not a design pattern; it’s a set of questions that we need to be able to answer about our content."

slide: takeaway

These notes are just the main ideas of the talk. They don’t contain anecdotes and examples. If you want to learn more, I would advise watching the talk on the GDC Vault.

The Toolsmiths

The Toolsmiths logo

I took these notes as part of our little “Book Club” for GDC Vault Videos The Toolsmiths #vault club

Koray Hagen is a member of the Toolsmiths community. The Toolsmiths are a community of Game Tool Developers that are passionate about improving the way people make games.

Join us on Slack.

Join us on Twitter.