Recommending Cryptocurrency Trading Points With Deep Reinforcement .

Appl. Sci. 2020, 10, 15063 of 18based on the price fluctuations, nor did they analyze the average size of the price percent increases andpercent decreases their models were predicting. More classical approaches of using historical pricedata of cryptocurrencies to make predictions have also been tried before. Hegazy and Mumford [22]achieved 57% accuracy in predicting the actual price using a supervised learning strategy. Jiang andLiang [23] utilized deep reinforcement learning to manage a bitcoin portfolio that made predictionson price. They achieved a 10 gain in portfolio value. Last, Shah and Zhang [24] utilized Bayesianregression to double their investment over 60 days. Fischer et al. [25] have successfully transferred anadvanced machine-learning-based statistical arbitrage approach. More relevant research to our paperhas been done by A.R. Azhikodan et al. [26] aimed to prove that reinforcement learning is capable oflearning the tricks of stock trading.Similarly, John Moody and M. Saffell [27] used reinforcement learning for trading, and theycompared trading results achieved by the Q-Learning approach with a new Recurrent ReinforcementLearning algorithm. The scientist Huang Chien-Yi et al. [28] proposed a Markov Decision Processmodel for signal-based trading strategies that give empirical results for 12 currency pairs and achievepositive results under most simulation settings. Additionally, P.G. Nechchi et al. [29] presented anapplication of some state-of-the-art learning algorithms to the classical financial problem of determininga profitable long-short trading strategy.However, as a logical continuation of previous attends to use machine learning for trading, wewould like to propose a deep reinforcement learning approach that learns to act properly in the stockmarket during trade and serves to maximize trader’s profit. Generally, an application works as follows.To start, it takes a random action a1 (for example, it buys) and moves to the next action a2 (sells). Afterselling, the reward is estimated by subtracting the selling price with the buying price r a2 -a1 . If theresult is positive r 0, an agent will get a positive reward. If r 0, the agent will be punished withnegative reward. Based on the value of the reward, the application realizes the quality of its actionsand uses it to improve the skill of how to take the right action.This paper proposes an application that recommends taking the right action and maximizeinvestor’s income by using a deep reinforcement learning algorithm.The paper comprises of 5 sections. Section 1 is a presentation that gives brief information aboutthe trade procedure, systems, and related researches to our topic, Section 2 is a background of thepaper that enables us to see how essential to utilize a deep reinforcement learning approach; Section 3hypothetically portrays the proposed application; the Section 4 examines the analysis procedure andresults; lastly, Section 5 finishes up our paper.2. BackgroundIn the introduction part, we briefly introduced five common active trading strategies. In thissection, we will try to dive deeper into them. For an explanation of how they are useful in real life, wedownloaded bitcoin hourly historical price data and tested them with some of these strategies. Indeed,any kind of historical data (minutely, hourly, daily, and weekly) are available on the web sources, andmost of them are free to use. The next step of the experiment is trading in the cryptocurrency marketby strategy rules’ guidance.2.1. The Double Crossover StrategyAs noted above, there are two kinds of price averages required for this strategy. The longercalculating period (in our case 60 days price average) is called the long-term moving average (LMA)and the shorter period called the short-term moving average (SMA—15 days price average). Tocalculate the simple moving average, whether it is ten days or 60 days, one adds the price of 10 daysand divides into ten or sums up the 60 days and divides into 60. For the next day calculation, drop thelast data and add the latest data point and recalculate the average. In this way, the average keeps on

The golden cross interpreted as a signal to a final upward turn in a market by analysts andtraders. So, the operation tactics can be “buy at the golden cross”. Conversely, a similar intersectionof the lower moving average is a deadly cross, and it is believed that it signals a decisive decline inthe market. So, it is a useful act like “sell at death cross”.Appl. Sci.10, 1506of 18We2020,downloadedhourly historical data of Bitcoin from 2 of October 2016 to 1 of the March42019year and traded based on the Double Cross strategy. The trade process visually described inFigure 1. Trade process went as follows: for starting, the trader’s capital was 10 000 cash and 20moving forward with every new data point being generated. Below, Equation (1) is a quick explanationBitcoin (BTC) coin, and he/she acted with the guidance of strategy: bought at the golden cross, soldof how to calculate ten days moving average, where d—daily price, p—points, n—number of points:at the death cross.d d ··· d9 d10 must dpay d ··· d10 d11 to is transactiond d ··· dn 1 dnThey are importantOne factor thatp1 every 1 2investor; p2 2 3 attention; . . . ; pn n 9 n 8 10 costs.(1)1010because it is one of the key determinants of net returns. High transaction costs can mean thousandsof dollarslost fromnotjust thenext,costswethemselves,butbecausethe costspoints.reduceTothefindamountof capitalAccordingto thestrategy,should findtheright crossovercrossovers,weavailableinvest[30].canuse a torulethat explainedin the strategy:Table 1 presents the main information about the process: invested money for trade includes sum1.Downtrendchangesto the uptrend,crossesover theand secondly,theof cashand valueof availablecoins at thethatpricemoment,numberof short-term,appeared goldenand deathshort-termoverfeethethatlong-term.is called“GoldenCrosspoints,sumcrossesof tradingis 0.15%Thisof thetradingthevaluetakenCross”.whenever trader acts, cecrossesunderthefinalshort-term,and secondly,after trading also includes sum of all cash and available coins value at thetrade rm.Thisiscalled“DeathCross”.of trade process, whether the capital is lost or gained profits. For evaluating the trading process, weshowThetwogoldencases crossin Table1. The he andtrader’sactinterpretedas aonesignalto a finalin a raderSo, the operation tactics can be “buy at the golden cross”. Conversely, a similar intersection of theholds movinginvestedaveragemoneyisandcoins cross,until themomentthatof tradingwithoutperforminglowera deadlyandfinalit is believedit signalsa decisivedecline tmentwiththefinalcapital.So, it is a useful act like “sell at death cross”.Accordingto Table1, thehistoricalquality loadedhourlydataBitcoinfromwhen2 of Octoberto 1Crossoverof the �action.Theseresultsindicateoneoftheweaknessesyear and traded based on the Double Cross strategy. The trade process visually described in Figure 1.of the processstrategy.wentHowever,the singleresult isthenottrader’senoughcapitalto judgestrategy.ForBitcointhat reason,Tradeas follows:for starting,wasthe10whole000 cashand 20(BTC)wewilltestthestrategydifferently.coin, and he/she acted with the guidance of strategy: bought at the golden cross, sold at the death cross.Figure1. 1.TheBitcoinhourlydatadiagramfromthethe2 of2October20162016to the1 of1March2019.2019.The TheDeadFigureTheBitcoinhourlydatadiagramfromof Octoberto theof Marchand GoldenCrosspointsmatchedwith greenand redpoints,Deadand GoldenCrosspointsmatchedwithgreenrespectively.and red points, respectively.One factor that every investor must pay attention to is transaction costs. They are importantbecause it is one of the key determinants of net returns. High transaction costs can mean thousandsof dollars lost from not just the costs themselves, but because the costs reduce the amount of capitalavailable to invest [30].Table 1 presents the main information about the process: invested money for trade includes sumof cash and value of available coins at that moment, number of appeared golden and death Crosspoints,sum of trading fee that is 0.15% of the trading value taken whenever trader acts, money after tradingalso includes sum of all cash and available coins value at the final trade process, quality of tradeprocess, whether the capital is lost or gained profits. For evaluating the trading process, we show twocases in Table 1. The first one shows how invested money changes when the trader’s act goes based onthe Double Crossover strategy’s guidance. The second case describes when a trader holds investedmoney and coins until the final moment of trading without performing any actions, then compares theinitial investment with the final capital.

Appl. Sci. 2020, 10, 15065 of 18Table 1. Main information about the trading process.InvestedMoney forTrading ( )22 223.4Trade withNumber of“Buy”ActionsNumber of“Sell”ActionsTransactionFee ( )Money afterTrading ( )Quality ofTrading (%)DC strategy1211111 822.373 779.1Grew 332.0holdposition00086 248Grew 388.0According to Table 1, the quality of the trade process when applied Double Crossover strategy is56% lower than trading only with the “hold” action. These results indicate one of the weaknesses ofthe strategy. However, the single result is not enough to judge the whole strategy. For that reason, wewill test the strategy differently.Continuously, to test the strategy more deeply, we divided our last prepared data into two parts.The first part involves the data where cryptocurrency price is wildly raising, and the second partincludes the data where the price is decreasing. The Double Crossover strategy was applied to bothparts. Table 2 is shown the main trading criteria such as prepared money for trading, the number ofappeared Golden and Death Crosspoints, all money spent for the transaction, money after the tradeprocess, and the column for the process evaluation.Table 2. Main information about the trading process for the first part (increasing period).Data(Part)Trade withDC strategyInvested Moneyfor Trading ( )“Buy”Actions“Sell”ActionsTransactionFee ( )Money afterTrading ( )Quality ofTrading (%)5440341.1450 529.2Grew 2027.3hold position000399 500Grew 1797.6DC strategy65701441.9105 611.5Lost 73.400086 486Lost 78.2FirstSecondhold position22 223.4396 500According to Table 2 information, during the experiment on the first part of data, there are 54Golden and 40 Death Crosspoint occurred. As a result of the strategy, the trader gets 20.2 times moreprofit, which is very well. Additionally, trade with the Double Crossover strategy brings a noticeable228% more profit than trade only with “hold” action. However, the experiment with the second part ofdata shows 70 “sell” actions and 65 “buy” actions, and in consequence trader’s money decreased by73.4%. Unfortunately, for this period, the strategy did not give expected results: instead of profiting,the trader lost more than 73% of the invested money, and it is 5% higher than if he/she does not usethe strategy.The above-given experiment shortly tells the strategy can be useful as it can be dangerous for thetrader’s capital.2.2. Day TradingDay trading is perhaps the most famous style of active trading. This is often considered thepseudonym of the most active trading. Day trading is the method of buying and selling securitieswithin the same day. Positions are closed on the same day they were occupied, and no position is heldduring the night. Traditionally, day trading is carried out by professional traders, such as specialists ormarket makers [31].Due to the nature of financial leverage and possible quick returns, daily trading results can varyfrom extremely profitable to extremely unprofitable, and high-risk traders can generate either hugeinterest incomes or huge interest losses [32]. Below are a few basic trading methods by which daytraders try to make a profit.-News playing. The basic strategy of playing a game in the news is to buy a stock that has justannounced good news, or a short sale on bad news [33].

Appl. Sci. 2020, 10, 15066 of 18-Rebate trading. Rebate trading is a stock trading style in which electronic communicationnetworks (ECN) rebates are used as the main source of profit and income [34].Contrarian investing. Contrarian investing is a market timing strategy used in all tradingperiods[35].Appl. Sci. 2020, 10,x FOR PEER REVIEW6 of 18Day traders are typically well-educated and well-funded. For day trading, there is no such kindmarketnews.the reason,nobuyneedtestThehistoricalprice aredatawith daystrategy.of specificruleThatthat isindicatesto theortosell.day traderslookingfor tradingany goodor isingstatistics[36]aboutdaytradingsuchmarket news. That is the reason, no need to test historical price data with day trading strategy.Inas:addition, researchers have discovered 24 very surprising statistics [36] about day trading such as:80%-80% ofof allall dayday traderstraders leaveleave withinwithin thethe firstfirst twotwo radeonlymonth.Withinthreeyears,Among all day traders, almost 40% of traders .continue day trading. Five years later, only 7% remained.Day traders with good past performance continue to make good profits in the future, althoughDay traders with good past performance continue to make good profits in the future, althoughonly about 1% of day traders can predictably make a profit minus fees.only about 1% of day traders can predictably make a profit minus fees.These statistics tell that day trading strategy might be dangerous for non-specialist traders.These statistics tell that day trading strategy might be dangerous for non-specialist traders.2.3. Swing Trading2.3. Swing TradingSwing trading is a trading methodology that seeks to capture a swing (or “one move”). SwingSwing trading is a trading methodology that seeks to capture a swing (or “one move”). Swingtrading has been described as a form of fundamental trading in which positions hold longer than onetrading has been described as a form of fundamental trading in which positions hold longer than oneday. Most fundamentalists are swing traders since it usually takes several days or even a week today. Most fundamentalists are swing traders since it usually takes several days or even a week tochange the fundamental indicators of a company in order to cause a sufficient price movement tochange the fundamental indicators of a company in order to cause a sufficient price movement to makemake a reasonable profit. Figure 2 graphically shows the basic terms of swing trading.a reasonable profit. Figure 2 graphically shows the basic terms of swing trading.Figure 2. The basic term of Swing trading.Figure 2. The basic term of Swing trading.In reality, Swing trading sits in the middle of the continuum between day trading to trend trading.reality,sits in themiddlethe continuumbetweenday tradingto atrendA dayIntraderwillSwinghold atradingstock anywherefroma few ofsecondsto a few hoursbut nevermore omafewsecondstoafewhoursbutnevermorea trend trader examines the long-term fundamental trends of a stock or index and may hold the stockthana fewa day;a trendtrader examinesthe long-termfundamentaltrendsa stockor indexand daysmayforweeksor months.Swing tradershold a particularstockfor a ofwhile,generallya ldaparticularstockforawhile,generallyto two or three weeks, which is between those extremes, and they will trade the stock based on itsa few days ortointra-monthtwo or threeoscillationsweeks, whichis betweenthoseandthey will trade the nsbetweenoptimismandpessimism[37].set of mathematically based objective rules can be used to create a trading l analysis to give buy or sell signals. Simpler rule-based trading strategies que of Alexander Elder, which tests an instrument’s market trend activity using three oving average closing prices. Only when the three averages are moving in an upward directionmovingaverage closingOnlywhenthe threemovingin anshiftupwarddirectionisisthe instrumenttraded prices.Long andonlytradedShortaverageswhen thearethreeaveragesdownward[38].the instrumenttraded Shortwhen theTheexperimenttradedand itsLongresultandwillonlybe discussedin Section4. three averages shift downward [38]. .Swing trading tips, like all investment strategies, are never to lock yourself into a specific set oftradingtips, likeIfallinvestmentstrategies,are nevertoforcelock yourselfintotoaridespecificofrulesSwingbut ratherguidelines.somethingis evident,tradersdo notthemselvesit outsetjustrules but rather guidelines. If something is evident, traders do not force themselves to ride it out justbecause the strategy dictates. Swing trading is a middle of the road investment strategy and shouldbe considered when developing a personal approach.2.4. Scalping Trading

Appl. Sci. 2020, 10, 15067 of 18because the strategy dictates. Swing trading is a middle of the road investment strategy and should beconsidered when developing a personal approach.2.4. ScalpingTradingAppl. Sci. 2020,10, x FORPEER REVIEW7 of 18Scalping is probably the fastest procedure utilized by dynamic merchants. It incorporates usingdifferentvalueaboutbid-ask spreadsandbidrequestThe methodologyby andlargeworks by makingtheholesspreadorbypurchasingat thepricestreams.and sellingat the askingpricethat ceandsellingattheaskingpricethatvaluegets the distinction between the two value focuses. Bid-ask spread is described in Figure 3. Scalpersgets the distinction between the two value focuses. Bid-ask spread is described in Figure 3. Scalpersendeavor to hold their situations for a brief period, in this way diminishing the hazard related to theendeavor to hold their situations for a brief period, in this way diminishing the hazard related totechnique.the technique.Figure 3. Make a profit between the bid and ask price difference.Figure 3. Make a profit between the bid and ask price difference.Additionally, a scalper does not attempt to exploit massive moves or move high volumes. Instead,Additionally,scalperadvantagedoes notattemptto exploitmassivemovesmoveadditionalhigh volumes.they struggle ato requireof littlemoves thatoften occurand movesmaller orvolumesSince theamount advantageof profits peroftradeis movestiny, scalpersexplorefor otherInstead, typically.they struggleto requirelittlethat nd the frequency of their trades. In contrast to swing traders, scalpers like quiet markets that areadditional typically. Since the amount of profits per trade is tiny, scalpers explore for other liquidnot at risk of fast worth movements so that they will probably build the repeatedly unfold on identicalmarkets bid/askto extendthe frequency of their trades. In contrast to swing traders, scalpers like quietcosts. Scalpers have different methodologies that can tell whether buying or selling. Mostly,markets theythatareareusingnot threeat riskof fastworthmovementsso charts.that theyprobablybuildAveragethe repeatedlydifferentmovingaveragesshort-termOne willof themcalled Movingunfold onidenticalScalpershavedifferentmethodologiesthat oncanwhether buyingRibbonEntrybid/askStrategy, costs.which isused 5-8-13simplemovingaverage combinationthetelltwo-minutechartto identifystrongcan bebought orsold.or selling.Mostly,theyare trendsusingthatthreedifferentmovingaverages short-term charts. One of themHowever, economists suggest not to believe positive things about scalping. There is no singlecalled Moving Average Ribbon Entry Strategy, which is used 5-8-13 simple moving averageverified formula that guarantees you scalping success in at least 90% of cases. Similarly, if somethingcombinationon the two-minute chart to identify strong trends that can be bought or sold.sounds too good to be true, it is most likely true, especially in an atmosphere of forex scalping.However,economistsnot to believeis no singleAs always,it is truesuggestthat the investmentwill bepositiveunder risk,thingsso forexabouttradersscalping.can benefitTherefrom doingverified theirformulathat guaranteesyou scalpingsuccessin advisorsat least before90% ofcases. Similarly,if somethingdue diligenceand/or advisingindependentfinancialengagingin ranges tradingorotherstrategies.sounds too good to be true, it is most likely true, especially in an atmosphere of forex scalping.As 2.5.always,it Tradingis true that the investment will be under risk, so forex traders can benefit fromPositiondoing their due diligence and/or advising independent financial advisors before engaging in rangesA trader who holds a position in an asset for a long period of time is called a position trader.trading orother strategies.The period can vary from several weeks to years. In addition to “buy and hold”, this is the longestretention period among all trading styles. Positional trading is largely the opposite of day trading.2.5. Positi

applied sciences Article Recommending Cryptocurrency Trading Points with Deep Reinforcement Learning Approach Otabek Sattarov 1, Azamjon Muminov 1, Cheol Won Lee 1, Hyun Kyu Kang 1, Ryumduck Oh 2, Junho Ahn 2, Hyung Jun Oh 3 and Heung Seok Jeon 1,* 1 Department of Software Technology, Konkuk University, Chungju 27478, Korea; otabeksattarov95@gmail.com (O.S.); azammuminov92@gmail.com (A.M .