High-Frequency Trading And Ultra Low Linkedin /in .

High-Frequency Trading and Ultra LowLatency Development TechniquesNimrod SapirVP R&DqSpark imrodsapir/

About me Been working in the HFT world for the last 5 years Worked on performance sensitive code for most of mycareer (mostly for storage systems)2

What am I going to talk about ? What is algo trading and high-frequency trading How does high-frequency trading impacts the market Main challenges in designing high-frequency tradinginfrastructure Development techniques used in qSpark for the worldof high-frequency trading3

What is algorithmic trading/High Frequency trading Algorithmic trading is any software which follow a predefined algorithmto place trading instructions High-frequency trading is algorithmic trading characterized with veryhigh trading rate and short investment horizon. Usually, HFT algos do not try to predict overall long term marketbehaviour (i.e. will it go up or down) HFT algorithm profitability is dependent on its ability to perform tradingactions at critical points in time in an extremely4latency-sensitive manner.

High-frequency trading market share (estimation) Although there is no official statistics, HFT is estimated to account toat least 50% of the US equity (shares) trading volume Notice that trading volume does not equal capital The market share of HFT has declined, as did profitability, since thepeak year (2009)5

High frequency trading market share (estimation)6

High frequency trading market share (estimation)VIX Volatilityindex. Reflects thelevel of anxiety in themarket.7

High frequency trading market shareDow Jones - 2017-20188

Is HFT good for the economy The questions on high-frequency trading impact on theeconomy is old as HFT itself Many concerns rose around events such as the “flashcrash” of 2010 Two known benefits of HFT are improvement of marketliquidity and narrowing of bid-offer spread9

Market making Alice comes to the market andwould like to buy apples Apples are widely available in themarket, and there are alwaysbuyers and sellers for them However, in order to get funds, sheneed to sell the dragon fruits shealready has - which are a not verypopular!10

Market making For actual trade to happen,alice has to hold her orderopen for 6 hours During this time she maydecide to give up (forexample, if price of applesrises and she can’t usethose funds anymore)11

Market making Carl, the trading algo iswilling to match Alice sell,allowing her to buy applesimmediately Carl will wait until Bobshows up and completethe transaction12

Market making Carl is considered a “market maker” as it enabled all the transactions,which may not have happened without it. Although it seems that Carl did not earn from the transaction, in thestock market it will actually earn rebates, which are percentages ofthe fees the market took from each transaction. However, Carl also took a large risk, as there is no guarantee that abuyer would come in, or that the price wouldn’t drop13

Wait, but what is fast? The definition of HFT is very vague, and is constantlychanging At the beginning of the 21st century - a turnaround ofseconds would be considered “high frequency” Today, measurements is in microseconds14

Wait, but what is fast? Light travels at 300 m/microsecond Forget about running your HFT rigfrom your own data center or fromthe cloud Also, the days of looking for theclosest location to the exchange areover - you must reside inside theexchange15

Market fairness and network latency Due to strict market regulation, anydifference in external network latencywas evened out. Therefore, the real competition nowresides inside the traders' technologicalstack16

HFT trading infra - main challenges.And they will lead you through the dark ot the widest, deepest river ofwealth ever known to man. You'll be shown your place on the riverbank,and handed a bucket all your own. Slurp as much as you want, but try tokeep the racket of your slurping down .- Kurt Vonnegut -17

HFT trading infra - main challenges A good trading product has the right balance of profitable trading logic,strong trading infrastructure and ability to quickly act and react tomarket events Nothing will save you if your trading logic is misguided or if you are slowto react - you live and die by your technical stack. Brand is non-existing - nothing prevents a smarter or quicker competitorfrom taking over your “market share” All actions must comply to strict market regulation - bugs can easilyresult in fines!18

HFT trading infra - main challengesWheretalkinga betteraboutreferencemay be something Whenthe competitionin thelike that:HFTworld, you may imagine something likethis:19

HFT trading infra - main challenges The market behavior can never be accurately predicted, and you cannever be certain you are going in the right direction You need to be able to be extremely quick, not only to make a quicktransaction, but also to be able to revert quickly and cut your losseswhen you are stuck with a bad trade You are never “fast enough”, every nanosecond you can cut of yourreal-time flow will result in increase in profitability, and vice-versa20

Main development approach in the qSpark trading infra End-to-end kernel bypass - system calls are too slow to use in real time Avoid context switching, queuing and data transfer between threads asmuch as possible Deterministic, static code flow, which makes as many decisions aspossible in compilation time Minimize cache misses and wrong branch prediction Use custom-tailored data structures for specific use cases It is not faster if you haven’t measured it21

Deterministic code flow and branching minimization Every run-time action has a performance penalty This is worsen by the fact in the case of branchmisprediction which means wasted CPU cycles Our design strives to create a deterministic, static flow,which minimizes runtime branching by moving overhead tocompilation and initialization time22

Deterministic code flow and branching minimization23

Deterministic code flow and branching minimization24

Compile time polymorphism using CRTP (Curiouslyrecurring template pattern)class order{virtual void place order() {// Generic implementation.}};class specific order : public order{virtual void place order() override{// Specific implementation.}};class generic order : public order {// No implementation};25

Compile time polymorphism using CRTP (Curiouslyrecurring template pattern)template typename actual type class order{void place order() {static cast actual type* (this)- actual place();}void actual place() { // Generic implementation }};class specific order : public order specific order {void actual place() { // Specific implementation. }};class generic order : public order generic order {.};26

Compile time polymorphism using CRTP (Curiouslyrecurring template pattern)template class Execution, template class A,class B class SocketHandlerType SocketHandlers::Tcp class BOE2 : public ExecutionProtocol Execution,SocketHandlerType, BOE2SequenceSourceType,HeartbeatPolicy::Send { . }27

Deterministic code flow and branching minimization There are many more techniques that can be used for achieving amore deterministic code flow Maps with static values which can be evaluated in compile-time Compile-time configuration which replaces runtime flags with templated values Rearranging and/or statements order to move the more predictable values first Of course: constexpr all the things! Those techniques may have their own cost - mostly in compile-time,but sometimes also in run-time (code bloat)28

Warming up the cache Cache misses are one of the highest overhead for a low-latencycode. However, cache is very unpredictable - in multi-threadedenvironment, there is a constant fight for the L3 cache This is worsen by the fact that the most critical flow issometimes extremely rare29

Warming up the cache When trigger actually occurs,the likelihood of the orderplacement flow to be in thecache is extremely low In addition, branch predictionwill assume order is neversent30

Warming up the cache Now the order placementflow is way likelier to be in thecache, and branch predictionis more balanced Sounds simple, but there aremany complications31

Warming up the cachesize t g total value{};void add order value(Order& order){g total value order.get amount() * order.get price();} There is a side-effect here, that we need to eliminate Naive approach, let's check if the order is warming only32

Warming up the cachesize t g total value{};void add order value(Order& order){if (!order.is warming)g total value order.get amount() * order.get price();} We may have made things way worse! Multiple mispredictions can easily lead to warming actually addingperformance penalty33

Warming up the cache34

Warming up the cachestd::array size t, 2 g total value{};void add order value(Order& order){g total value[order.is warming] order.get amount() * order.get price();}size t get order value(){ returng total value[false] }; The misprediction is eliminated Although we are “warming” the wrong entry in the array, locality makesit very likely that we are actually warming both35

Warming up the cache This is a very simple example, but actually avoiding allside-effects in a complicated flow, without skipping any part ofthe code, may be very challenging and may require majorredesign Any bug here would (and did) lead to serious issues Therefore, handle with care! That said, in the world of micro optimization, cache warming isextremely efficient!36

Tailor-made data structures for specific use cases Our code contains many data structures which are optimizedfor specific use cases Some are extremely general and complex, some were madespecifically to resolve specific issues Here is one simple example: static flat map37

Static Flat Mapvoid foo(std::map . & multi threaded small map,lock type& very busy lock){std::lock guard . guard(very busy lock);for (auto& item : multi threaded small map){.}}38

Static Flat Mapvoid foo(std::map . & multi threaded small map,LockType& very busy lock){std::map . local map;{std::lock guard . guard(very busy lock);local map multi threaded small map;}for (auto& item : local map) {.}}39

Static Flat Map Keep a sorted array Use binary search to find items Result: Much better performancefor copying and iterating over asmall map with known size40

Static Flat Mapvoid foo(static flat map . & multi threaded small map,lock type& very busy lock){static flat map . local map;{std::lock guard . guard(very busy lock);local map multi threaded small map;}for (auto& item : local map) {.}}41

Static Flat MapCopy Timestd::mapStaticFlatMap 25usec1usec 42

Static Flat MapPros: Quick iterationQuick copyQuick lookup and editSequential and StaticCons: Slow insert and removeLimited - size must be known in advanceNot as good for large mapsNot as good for large 43

Performance measurement When it comes to micro optimization and ultra low-latency, noguarantee for “better average performance” is acceptable as-is. This means that each performance tweak has to becontinuously measured to show better performance. For example, -march and -mtune flags actually degraded performance in ourenvironment. However, testing the entire matrix of possible combinations ispractically impossible.44

Performance measurement45

Performance measurement46

Performance measurement This performance measurement technique is nice, but it has alot of overhead The minimal overhead is simply taking a timestamp. Forexample, in our environment:Timestamp accuracy150 nanosecond1 microsecondTimestamp taking overhead50 nanoseconds10 nanoseconds We have to separate lightweight real time counters fromintrusive measurements used in production47

Disclaimer Premature optimization is the root of all evil (Donald Knuth) Premature micro-optimization is just plain stupid! The techniques described all have very serious costs, pitfallsand trade offs Use with care, and only when micro-optimization is required48

We are hiring C com/in/nimrodsapir/Static Flat NKS!Any questions?

Algorithmic trading is any software which follow a predefined algorithm to place trading instructions High-frequency trading is algorithmic trading characterized with very