KYS before teaching your AI

In this article you will learn how many opportunities there were in the past 12 months to make profit on BTC and how many opportunities to avoid loss :)

Know Your Signal before teaching your Neural Network. Your are the teacher for AI, so you need to know the subject you are teaching, right?
AI is the student here. What is AI? It tends to be shown in a context where AI = Super Intelligence. Well - it’s not. It’s what the name says - it’s Artificial Intelligence, it is mechanical, a learning engine. Engines are always amazing tools. It can be super when applied correctly in some context, but only if someone will use the tool (=teach) correctly. You might have heard about Microsoft racist AI? https://en.wikipedia.org/wiki/Tay_(bot)

So before I start learning neural network, I need to do my own lesson and learn a bit about my time series also TO UNDERSTAND WHAT I’M REALLY SEARCHING AND PLANNING TO DO.

Signal is a time series - a parameter that changes in time. My goal here is to find if it can be predicted and used to generate profit. There are numbers of hidden assumptions in the previous statement:

1. Given time series value can be predicted - I assume it’s true, but is it? Here we can go even deeper and ask what is true? In science often there is no 100% truth, just probability of events. Here it is good enough. If the time series can be predicted in 60% of cases this could be good enough to generate profit.

2. What is profit? Time series changes all the time, up and down. By profit we could say 4% growth or 10% growth. But it’s not precise at all until you set the time frame. Is 4% in a year good enough? For sure not for me. So maybe 4% in a week? But why in a week not in 5 days? Why 4% , not 9%? - A VALID QUESTIONS TO ANSWER.

So I have found that I do not know something basic - what kind of profit is important for me and what kind of profit I could get.
Name it short - what is the dynamic of significant changes in the signal. How often does it go up and down during period of 1 year. Dynamics is not only how often it changes significantly, but also how rapid is the change.

My next step is to find how often price changes significantly and how rapid changes are. Since I need to define significant growth or decrease I will take some nice numbers like 3, 5, 8, 13, 21.

After I get the answer I will not only know what I should search next, but also how to mark my signal to train the neural network later. There is many ways to teach, not many good ways though.

So I did a quick script to answer my question and found one more important thing to precise - it’s better to search for unique significant change, as these changes may overlap, as you can see on the picture below:

To have unique changes I decided to take the shorter of overlapping two while moving forward.
Modifying script to make it efficiently took me few days. I’m new to python; initial script was taking a very long time (like a week for only 3% :) to find unique changes using my simple algorithm. Trying to optimize it and improve my python skills took few days, after which it went down to 6 hours for 3% growth (still not amazing but good enough for me) and overall it took a week to have all numbers up and down.

So here is the result:
This set of chart shows how rapid is change split by percentiles to have distribution. Red are drops, green - ups. The outcome is dynamics of growth and decrease are similar in order of magnitude with acceptable differences and I may decide to use 90 percentile change time for my AI later.

The same result may be show also as table. For some people it may be confusing as there are 2 kinds of % values - to indicate growth type and to indicate how many results match specific dynamics. The table basically shows how much time you need to wait to have for example 5% growth. In 10% cases it will happen just in 4.4 minutes. Median for 5% is 52.6 minutes.

Percentile	+3%	+5%	+8%	+13%	+21%	-3%	-5%	-8%	-13%	-21%
10	1.6	4.4	12.5	42.7	202.9	1.9	5.7	17.6	52.9	181.7
20	3.4	10.7	34.1	88.2	358.7	3.8	12.3	32.8	92.8	520.4
30	6.4	20.9	70.7	175.6	961.9	6.5	20.0	53.8	189.0	867.4
40	11.1	36.2	106.0	267.6	1429.5	10.3	28.8	78.6	298.0	923.0
50	16.8	52.6	163.5	484.2	1502.1	15.3	45.4	125.5	591.3	1154.8
60	27.9	85.2	266.6	692.7	1904.6	23.2	66.5	179.6	731.3	1582.0
70	44.9	144.5	469.5	1232.4	2871.2	35.1	105.3	281.0	1053.5	2052.6
80	74.4	235.3	821.0	1858.4	3571.4	52.4	167.4	426.6	1314.8	2924.0
90	153.5	482.8	1297.0	3788.3	6586.2	99.8	326.1	1081.0	2727.3	6411.1
100	1974.1	2246.0	4382.2	10301.1	9068.4	1094.2	4107.9	5492.0	4704.8	11202.1

Also I see estimation of how many chances there are to make profit or to avoid loss.

+3%	+5%	+8%	+13%	+21%	-3%	-5%	-8%	-13%	-21%
1556	593	230	102	34	1390	513	175	65	24

It means for example there were 1556 spots of unique growth of 3% last 12 months. Another example there were unique 24 drops of 21% to avoid.

Seems that make sense to move forward.