Deterministic Simulation

Software Architecture of an MMO

David Salz

david@sandbox-interactive.com

Who am I?

 David Salz

 15 years in the industry

 CTO, Co-Founder of Sandbox Interactive

 35 people

 based in Berlin

 we only do Albion Online!

In this talk:

 What is Albion Online?

 Middleware + Design Decisions

 Server Architecture

 How we use Unity (and how not)

Albion Online

 Sandbox MMORPG

 Cross-Platform (Windows/OSX/Linux/Android/iOS)

 One World (no „Shards“ or „Servers“, not even for different platforms)

 Player-Driven Economy (everything is player-crafted)

 No Character Classes („you are what you wear“)

 Strong focus on PvP + Guilds

 Hardcore („Full Loot“ in PVP Areas)

 Pay-to-play model

 4 years in the making

 Currently in Closed Beta w/ 80.000+ „founding“ players

 „Release“ in Q4/2016

Albion Online

 The initial pitch (2012)

 Play-crafted equipment, buildings

 One World (like EVE Online)

 Guild vs. Guild Territotorial conquest

 Top-Down perspective + Combat (like League of Legends)

 Simple graphics, „can even be 2D“

 PVP focus, no PVE, no Mobs, no Dungeons

 Release in 2013 

Middleware Selection

 Engine Selection

 Unity made sense – inexpensive, accessible

 Cross-Platform was a „target of opportunity“

 Database Selection

 One world  need a very scalable database, NoSQL

 Cassandra sounded suitable

 still use SQL for query-heavy problems

 Networking Middleware

 Photon!

 can use C# on server (like in Unity!)

 works with all Unity platforms

Apache Cassandra

 NoSQL Database

 Originally developed by Facebook

 Open Source (Apache-License)

 Java

 Concept: everything is a hash table

 in-memory (as much as possible)

 optimized for high throughput

 scales horizontally (just add servers)

 redundant (no single point of failure)

 CQL: SQL-like language (w/ massive restrictions – it‘s NoSQL)

Apache Cassandra

 funny story from public Alpha Test…

 all buildings disappeared during lunch break

 found this in Cassandra Changelog:

 W(

 lesson learned: careful with cutting edge technology

* Fix possible loss of 2ndary index entries during

compaction (CASSANDRA-6517)

Apache Cassandra

 Cassandra uses timestamps to resolve

conflicts between database nodes

 also requires exact time sync beween servers

 i.e. ntp service

Hosting

 Hosted with Softlayer

 good, but quite expensive

 bare-metal machines only

 currently: 8x 10-Core Intel Xeon, 32 GB RAM

 Should be able to handle ~15k CCUs

 Originally:

 Hosted across 2 Datacenters (better ping for players)

 Worked, but bad idea! (inter-server disconnects, latency)

Networking

 Photon

 UDP-based protocol (reliable/unreliable as needed)

 TCP for chat, inter-server communication

 use only basic message functionality (no Photon frameworks)

 mostly simple messages, in rare cases JSON content

 Had to implement secure TCP-reconnect abilities

 Chat

 Tried IRC…

 … was only trouble (difficult to secure, customize)

 Implemented own system in a couple of days

Consequences

 Server needs to work without Unity

 Ideally, client works without Unity, too!

 think: tools, stress-test-bots!

 Use Unity only as rendering front-end

 cannot rely on Unity features for basic functions!

 levels / game objects

 collision

 pathfinding

Separation

ObjectFactory ObjectViewFactory

Object

+Position

+…

+RequestAction()

ObjectView

+Renderer

+AnimationCtrl

+HandleInput()

destroyed

changed

…etc..

created

Object

+Position

+…

+RequestAction()

Server Client Unity-Client

Interest-Management

changed

…etc..

obj-enter

obj-leave

Server

Server Farm

Game Server

World Server

Marketplace Server

GoldMarket Server

Statistics Server

Ranking Server

BackOffice Server

Chat Server

Client

Connect to different game

server depending on

location in game world

Game DB

(Cassandra)

Accounts DB

(Postgres)

Market DB

(Postgres)

GoldMarket DB

(Postgres)

 Todo: screenshot of gold market

 Todo: screenshot of marketplace

Game Servers / World Server

 game world split into unique „clusters“ of ~ 1 km²

 current world: ~600, next world: x2

 distributed among game servers

 player changes servers when travelling

 handoff done through database (even on same server)

 Instances are dynamically created

 world server coordinates this

 World Server

 responsible for everything „global“

 guilds, parties, coordination of guild battles etc.

Threading Model (Game Server)

IO Thread

Network IO

Thread

Event

Queue

Logging IO

Thread

Event

Scheduler

Cluster Thread

Event

Queue

Event

Scheduler

Cluster Thread

Pathfinding

Thread

Database IO

Thread

Cluster Thread makes

request

Events / Results get

put into Event Queue

Cluster polls Events

from Queue and

Scheduler

Threading Model

 Game logic for one game area („cluster“) is single-threaded

 Interactions between game objects too complicated otherwise; would

require lots of locking

 Objects „sleep“ whenever possible (especially mobs)

 All IO (Network, Database, Logging) and other „heavy lifting“

(pathfinding) offloaded to worker threads

 Instances (Player islands, instanced dungeons) are combined in thread

pools

 Other servers (chat, rankings) are completely multi-threaded

There is no spoon!

 There is no game loop!

 i.e. no Update() or Tick() functions for game objects

 Gameplay thread processes events in order of arrival

 Events are:

 Network input, i.e. player commands

 Timers (Event Scheduler)

 If an object needs a tick (Mobs do), it creates a recurring timer

 Results from DB queries (asynchronous)

 Results from pathfinding

 …

Interest Management

 Players should only „see“ (network-wise) what‘s on their screen

 important to reduce traffic…

 … and prevent cheating!

 Find objects a player can „see“ (and update when things move)

 Objects send messages to all players that „see“ them

 Needs to be efficient!

 ~500 mobs / cluster

 > 10.000 interactive objects (mostly trees )

 up to 300 players

Interest Management

grid based hash (10x10m cells)

● cells contain list of objects inside

● list is updated by object itself when moving

● cells fire events:

● ObjectEnteredCell()

● ObjectLeftCell()

player

● objects also fire events

● ObjectMoved()

● ObjectDestroyed()

● EventFired()

Interest Management

player

Interest area

Moves with player

Subscribe to objects when they

enter inner area

Unsubscribe when they leave

outer area

Tell player when:

● an object enters / leaves interest area

● an object in the area fires an event

Logging infrastructure

Game Server

Log File

Logstash

Agent

Redis

Elasic Search

Kibana

 track errors, generate statistics

 classic ELK stack (Elastic Search, Logstash, Kibana)

 kind of unreliable for us

 Also use Cassandra Counters a lot!

Logstash

Agent

Logstash

Agent

Own tools

Kibana

Client

Level Design

 Zones built in Unity (Unity scenes)

 Consist only of Unity prefabs („tiles“)

 ~ 30.000 / level

 Collision is painted for each tile

 Level gets exported to XML format

 level loading independed from Unity (for server, client)

 in Unity: instantiate prefabs

 actually faster and more controllable!

Level consists of

„tiles“ (Unity prefabs)

Tile has 3d collider

(for mouse picking +

ground height

definition)

2d collision is painted

per tile

(blocks movement /

blocks shots / blocks

placement etc.)

Ground tile

Object (non-ground) tile

Collision

 TODO: pictures, demo

Collision of all tiles

is blended together

Unity Features we (don‘t) use…

 Do:

 Editor: cool and customizable (but often too slow)

 Sound: yes, but want to switch to Wwise or FMOD

 Mechanim: now, after a long struggle (memory usage!)

 Not or very limited:

 Scenes – only to load GUIs (but not levels)

 Collision/Physics – only raycasts @ ground

 GUI – use modified NGUI instead

 Network – use Photon instead

 Baked light – scenes way too large

Unity troubles

 funny story…

 64 bit editor came just in time for us

 32bit editor was no longer able to build our project

 used > 2.7 GB RAM during build with is the max for

win32 processes

 headless was still working!

Character rendering

 Character + Equipment gets baked into one mesh

 want 1 draw call / character (well, + special effects)

 real-time

 parts of character mesh hidden depending on equipment

 Only one material for character + all equipment items

 Only one texture; also use vertex colors, specular

 Limit number of characters drawn (esp. on mobile)

 Todo: characters picture

Debugging

 Can run and debug complete environment locally on developers

machine

 Human-readable network log file

 Server and client errors caught in log files

 Searchable, with context

 Public staging system

 Community can preview patches, updates

 QA freelancers from the community

 Frequently mirror live database to public staging system

 Can download live database to developer machines

 TODO: screenshot of network log file

Cheaters

 people started using memory debuggers and packet sniffers

pretty much on day 1

 Absolute server authority is king!

 Includes what information you reveal to client!

 only one serious cheat ever found

 mistake on our side –“believed“ data from client without proper

check

 Gold Sellers / Account Stealing

 adding email verification greatly improved this

Cheaters

 .NET code very accessible for analysis

 camera hacks (minior problem because of interest management)

 found internal tools , cheats in code (not working on live)

 extracted data from client

 maps, player rankings

 cool community projects!

 Users built bots directly into client 

 Difficult to prevent

 Obfuscation – helps, but not much

 We are doing more integrity checks now!

 Future: Unity IL2CPP ?

Future Challenges

 Growing player base…

 Want a more diverse world

 exploring more modular level design right now

 Keeping iOS / Android version in sync with server

 ability to download data files without patching

 clients support old server / server supports old clients

Thank you!

Questions / Comments?

david@sandbox-interactive.com

We are hiring!

https://albiononline.com/en/jobs