Mike Fishy and Automation System Programming (Bots)


#344

Here’s the 24 hour results that yesterday’s bot accomplished:
20181204-Bot
We’re moving forward!

Just updated bot with today’s code and will report results!


#345

I was wondering when you would switch to BTC, I felt so dirty with USDT :money_mouth_face:


#346

I push every new version of the script to the repo.
I also created an “examples” folder with the firsts versions you posted, so people can learn with small chunk of code.

But I never coded in php before so I don’t know what’s the typical php directory structure ?

To all : Don’t hesitate to submit a pull request if you found a bug or made a significant change to let everyone benefits from your changes.

reminder for new people, here is the link to the repo : https://github.com/blombard/the-bitcoin-pub-bot
(You can DM me, if you have no idea what to do with it)


#347

My handle is pieman64

With the $3 MCU development board version, users can change the settings on the fly.

Latest screenshot which is a hybrid of the original ICXUSDT and the latest version before maximum orders was added. It also includes a 0.5% SL.

Did I forget to mention it works on iPhones too?

Today I’ll be mainly hacking the RSI formula :slight_smile:


#348

@Blynker

Truly amazing… really nice one. So are you able to explain how we would be able to put it on a phone?

This is very interesting. Major kudos.


#349

@Crash101 I don’t want to bog down the thread too much with how the smartphone system works.

But in essence you install an existing app that is already in the Apple / Google stores and create an account on a 3rd party server. You then scan a QR code image that I provide and the project is then “running” on your phone. At this point you will see the screenshots but without any data.

You have to flash the $3 MCU WiFi development board with the C++ file we are building and you are then good to go.

Send me a DM to let me know if you are an Apple or Android user and I will send you further details.


#350

@Mike_Fishy, let me ask the obvious (or not so obvious) question:

Are stop loss protections a handicap (both programmer’s mental model and code’s effectiveness) for automation purposes?

If I’m following your latest commentary correctly, here’s what I’m observing:

  • stop/loss thresholds interfere with the bot naturally managing the price action flow through other metrics (price action and RSI).
  • stop/loss effectively never happens if bot logic coded correctly to exit positions when a reversal and downtrend begins.

So I’m thinking stop/loss protection is the wrong approach altogether and is better suited to long-term hold strategies. After all, if we start tracking RSI, and I’m assuming, we’re talking about two, perhaps three period trends just like price action averages and we then add code to exit a position any time the RSI turns past some threshold and/or the price turns, then we’re effectively always exiting a losing position relatively early on.


#351

RSI is calculated from which trend?
If RSI is trending down, probably not a good one to hold right?
Mind you, if the RSI was trending down, we probably wouldn’t have bought it - right?

Have not added RSI Trend in yet in above code, but think about it :slight_smile:

RSI is calculated from the long trend average or this is how we have done it. If the short / medium / trend are all pointing up, this is effectively saying that RSI is going to be “high” as well. Therefore we are really only choosing trades that have the right preconditions to be executed. There are lots of trading opportunities and we have to just focus on the bot picking one that works for certain.

This is my take on your comments.

Thanks @Mike_Fishy / @mwlang / @Blynker for your great comments. I have got my head around that RSI code now. What had me most confused was the lack of variable declaration. I think I will be a happier person when we start to programme with C++.

On that topic, what should be download to start to compile into C++? Just so I can start messing around with C++.

Edit: Looking at the code, RSI function is being used to determine if we have reached a peak or a dip. So we do need it.

Crash


#352

@Mike_Fishy please help us on the RSI. Clear to me you want to buy at lowest RSI value possible. This is related to LT trend. As long as price is decreasing, rsi will decrease. As soon as the MT trend starts increasing, after a couple periods (depending on your settigs for ST MT and LT) this will also lead to an increase in the LT trend. So you are looking for ST up, MT up, RSI up. Please correct and explain to us how this works man! :slight_smile: :slight_smile:


#353

@user93. I dont think we need RSI as it is a function of the other functions. From a Bot’s perspective it is making an executive action based on the performance of LT / MT / and ST. If all are pointing in the correct postiion then this is a good point to be considering a buy order. If they are all pointing in the right direction this would be a good time for a sell order. The value of RSI really doesnt matter as we are just looking for the right opportunities to trade. As this bot will be looking at these values all the time, it will only pick trades that meet these conditions which also will be in line with trades that are of a high chance to be filled and to go through. The question then is reached what values to take for the LT MT and ST for each coin.

Given that most of the coins follow BTC price action then we will get a similar price action if we consider multiple BTC trends at the same time. It more or less is the same overlying trend for all the coins.

Do I have this right MIke? Anyone else have any thoughts?

Edit; Looking at the code RSI is picking up the peaks and the dips so it is an indicator that tells us of a trend reversal.

Crash


#354

Hi Team,

Good to see it’s got you thinking.

Firstly, there does need to be a stop loss, the hardest part is knowing when you should just call it quits on a trade as it is preventing funds from being used on other trades.

Secondly, the rules for entering a trade is still too simplistic, which is why the current code is entering into positions that are not good.

Thirdly, the RSI is a function of trade action, rather than trend. It is a calculation on how many consecutive shorts are done over how many consecutive longs. If there is more shorts than longs, it is lower and vice versa. The RSI should be something you want to check as a trend. Was the last RSI more or less than the previous RSI? If the RSI is trending down, even though the short term and medium term price is trending up, it means over-all, the CDA is probably going to go down in price over a longer period. What we really want to be looking for, is the short and medium terms price going up, with a low RSI that is now starting to increase. Then we should also be checking for Volume, which still needs to be added. Because if all those conditions are right, but the volume of trades is trending down (less volume in trades), then it is still likely the CDA is going to go down in price.

Just by playing with this code and seeing what it does and how it does it, understanding all the mistakes it is doing and what needs to be done - has already turned you all into better traders :slight_smile:

Edit: To Do list.

  1. Change the ST, MT and LT to be specific to each CDA separately. This facilitates the bot changing the trade cycle to be appropriate to the market for that CDA. We really shouldn’t be guessing the ST, MT and LT numbers ourselves, the bot should be doing it.

  2. Record the time for the last trade cycle. We track a CDA from it’s low to it’s high and back to it’s next low over the period to see how long it takes, so we can use this timing to guesstimate the time it takes for a trade cycle and adjust that specific CDA’s ST, MT, LT to fit that period of time.

  3. After entering a trade, track the gain or loss trend to determine the best time to exit the trade. Knowing when to exit is a lot of rules, we don’t want to exit too early or too late.

  4. Add RSI for ST, MT and LT, not just LT. We should have all 3 and use them to see what the short and medium trend is over the longer trend.

  5. Track how rapidly the price is going up or down in a time period specific to that CDA. If it trends up rapidly, it might be going through a pump and dump or it might be on a bull run or being dumped or there might be other shenanigans going on. All of which puts the CDA into a risk category where we should probably steer clear of it for a while.

  6. Check the 24 hour high and lows, how close is the price to the 24 hour high or low. The closer it is to a 24 hour low, the more likely, given the right market trend, it will head back to it’s 24 hour high.

  7. Check the percentage gain/loss on the 24 hour statistics, the higher the number, the more risk that CDA presents. We should be looking at ones that are reasonably stable with a range between -8% to +10%, if a CDA has had -30% or +40% - these are ones we probably want to avoid. We want to achieve regular good trades at reasonable percentages.

  8. Add Volume using the Depth API call.

  9. ST, MT and LT become dynamic values based on market activity, which actually tends to work best as a function of the Base CDA we are trading. For example, if we are trading the BTC to ALTS and back, we track the cycle of BTCUSDT to form the initial function that helps to determine the values for the BTC to ALTs periods.

This kind of needs to be done, tested, tuned and iterated on, before we even start with Ordering and all those rules as well.

Stay Fishy


#355

Homework…I will give it a go!!

Crash


#356

Hi Team,

We are also probably getting to the point where tasks need to be split out.

We should have a data collector task that is standalone and just collects data and stores it.

The question being, what should we store it in?

This could start getting to the issue of what platform we want to run on.

Personally, I use MongoDB, but if we really want to keep it platform independent, maybe just files, with an individual file per trading pair?

The data collector needs to be able to collect data store it and delete data older than a configurable period. Some of us might only want to keep data for a few days, or maybe a month depending on disk space, speed and so on.

Anyways, more to think about, as we really are now at the point where we should be starting to split out the tasks. With data collected 24/7, the replay feature will be easier, as it can just use previously stored data. Secondly the bot startup time is faster, as it doesn’t need to populate arrays before trading, as the data will already exist.

There are many advantages of splitting out the tasks and I think we are at that point where we should start doing it and decide how we want to achieve it. I’m leaving it up to you guys, but as I said, I personally use MongoDB, but even flat data files is probably suitable to allow people to experiment on different platforms. With development boards, storage might be an issue, they tend not to have much storage capacity, but if you can use a TF Card, you would probably be ok.

Stay Fishy


#357

@Mike_Fishy how much data per day would you expect the collector to generate?

The dev boards potentially have an inbuilt storage area of 15MB.


#358

I vote for Redis. Extremely fast key/value store and easy to access from pretty much any language/platform. Also, I already have much of the Ruby version of the code to pull data from Binance and cache in Redis using websockets rather than the RESTful API. This is code I’m willing to open source and maintain in the interest of speeding this particular project along.

Oh, and the saving to files will get quite slow as the pairs grow. I already went down this road a bit myself before switching to Redis. So, just something to keep in mind.

Another way to build for “cross-platform” is to dockerize the project and it may make the most sense to do this from the get-go so people can get up and running quickly locally and start contributing, too. And then for “production” can quickly establish a VPS server anywhere in the world and fire this thing up.

Thoughts?


#359

Optimus Prime!

I am not a coder by any means, so I Googled the difference between Redis & MongoDB for everybody’s benefit:
Redis-vs-MongoDB

…and finally, here’s yesterday’s bot results which generated a BTC-prices.txt file size of 224,748 KB:
20181205-Bot

Staying Fishy and reviewing home work…


#360

@Garzaro are you still at your PC?


#361

Yes sir I am at my PC


#362

Smartphone post (please skip if it’s not your thing).

We have people that are running the early beta version but without MCU’s.

Basically it just gives you a “look and feel” experience of what might be possible. It’s a quick and painless process to get the beta “running” on your Android or iPhone smartphone.

You can play around with the sliders, RSI, max orders and top of the list, the url link to this thread works. The way the building block widgets work means you can customise the app to your own design, move widgets, resize them, change colours and change parameters etc.

Memory: I believe Mike’s full bot runs to about 800MB of code. The MCU’s we use are 16MB devices but only 1MB is available for code, the other being available for data storage. Arrays can be held in the 15MB area but it would slow down data processing. Of the 1MB programmable memory almost all of it is already allocated to WiFi and SSL routines.

After including SHA-256 to sign each live Binance trade we are left with about 40kb of memory. Yes 40kb, so the smartphone version is never going to replace Mike’s 800MB version. One saving grace is that most of what you see in the smartphone app doesn’t require any memory as it’s purely processing data between the smartphone and a cloud based server.

Security: the code I have for the MCU is using SSL and SHA-256 etc but it’s important “keys” are not exposed as it would put your funds at risk. Binance has various security features implemented and we have some of our own. I’m not going to go into detail at this stage but it’s an important issue moving forward.

One long term possibility might be to interface Mike’s 800MB system with a smartphone via an API etc. Maybe I should create a separate thread for the “smartphone bot” with a link back to this thread.


#363

Yesterday when I wrote this I didn’t expect to still be working on it 24 hours later!!!

Took me some time to translate the PHP to C++ and my code was buggy for quite some time. What I found was that my MCU was struggling to get the correct result of summing some floats.

For testing purposes I drop the trend numbers right down to 5, 9 and 12 rather than 50, 90 and 120. After adding lots of debugging code I got the correct results as shown in the log below:

icx_loss: 0.00000000
losstotal: 0.00050001
icx_loss: 0.00000000
losstotal: 0.00050001
icx_loss: 0.00000000
losstotal: 0.00050001
icx_loss: 0.00000000
losstotal: 0.00050001
icx_loss: 0.00000000
losstotal: 0.00050001
icx_loss: 0.00000000
losstotal: 0.00050001
icx_loss: 0.00000000
losstotal: 0.00050001
icx_loss: 0.00000000
losstotal: 0.00050001
cdarsloss: 0.00004167
cdarsgain: 0.00004167
cdarsloss: 0.00004167
cdacalc: 0.00050001
RSI: 50.00000000
ICXUSDT	V:0.236100	  ST:0.235850:UP	  MT:0.235850:UP	  LT:0.235683:UP	  SPREAD:0.211775%	  RSI:50.000000	S:Buy Prep
[258153] Free RAM: 27720
====================================
Iteration = 40
Running Time: 4.32 mins 
Current running percentage = 0.000
====================================

Most of the time it simply wouldn’t come up with the correct answers.
Currently RSI is just working on lt and with settings of 3, 4, 40 for the trend values it comes up with the correct results. If I change the trends to 3, 5, 12 it picks a coding bug and fails to correctly sum the floats.

Note this is with dropping the size of the lt array from 40 to 12 and only increasing the mt from 4 to 5. Furthermore mt is not used in the RSI calc. So it’s not a general out of memory issue as the working code has much bigger arrays overall.

Currently I am thinking that elsewhere in the code the mt array is going out of bounds and bleeding into the lt array, thus messing up the float sum. It’s strange though that it doesn’t seem to go out of bounds when mt = 4.

Off to do more bug hunting!!!