Custom Strategy Tester based on fast mathematical calculations 27 February 2018, 13:31
Introduction
The Strategy Tester provided in MetaTrader 5 has powerful features for solving a variety of tasks. It can be used to test both complex strategies for trading instrument baskets, and single strategies with simple entry and exit rules. However, such a vast functionality does not always prove useful. Often, we just need to quickly check a simple trading idea or make approximate calculations, the accuracy of which will be compensated by their speed. The standard tester in MetaTrader 5 has an interesting but rarely used feature: it can perform computations in the math calculations mode. This is a limited mode for running the strategy tester, which nevertheless has all the advantages of full-fledged optimization: cloud computing is available, a genetic optimizer can be used and it is possible to collect custom data types.
Custom strategy tester may be necessary not only for those who require absolute speed. Testing in the math calculations mode opens the way for researchers as well. The standard strategy tester allows simulating trade operations as close to reality as possible. This requirement is not always useful in research. For example, sometimes it is necessary to obtain an estimate of the net efficiency of a trading system, without taking slippage, spread and commission into account. The math calculations tester, developed in this article, provides such ability.
Naturally, one cannot square the circle. This article is no exception. Writing a custom strategy tester requires serious and time-consuming work. The goal here is modest: to show that with the right libraries, creating a custom tester is not so difficult as it might seem at first.
If the topic proves interesting to my colleagues, this article will see a continuation that develops the proposed ideas.
General information about the math calculations mode
The math calculations mode is launched in the strategy tester window. To do this, select the menu item of the same name in the drop-down list:
Fig. 1. Selecting the math calculations mode in the strategy tester
This mode calls only a limited set of functions, and the trading environment (symbols, account information, trade server properties) is not available. OnTester() becomes the main call function, which can be utilized by users to set a special custom optimization criterion. It will be used alongside other standard optimization criteria and it can be displayed in the standard user strategy report. It is outlined in red in the screenshot below:
Fig. 2. Custom optimization criterion calculated in the OnTester function
The values returned by the OnTester function are selectable and optimizable. Let us demonstrate this in a simple expert:
//+------------------------------------------------------------------+
//| OnTesterCheck.mq5 |
//| Copyright 2017, MetaQuotes Software Corp. |
//| http://www.mql5.com |
//+------------------------------------------------------------------+
#property copyright "Copyright 2017, MetaQuotes Software Corp."
#property link "http://www.mql5.com"
#property version "1.00"
input double x = 0.01;
//+------------------------------------------------------------------+
//| Tester function |
//+------------------------------------------------------------------+
double OnTester()
{
double ret = MathSin(x);
return(ret);
}
//+------------------------------------------------------------------+
Its code contains nothing but the input parameter x and the OnTester function, which calculates the sine value from the passed argument. In this case, x. Now try to optimize this function. To do this, select the "Slow complete algorithm" optimization mode in the strategy tester, and the previous simulation mode: "Math calculations".
Set the variation range of x in the optimization parameters: start — 0.01, step — 0.01, stop — 10. Once everything is ready, run the strategy tester. It will finish its work almost instantly. After that, open the optimization graph and select "1D graph" in the context menu. This will show a sine function in the graphical interpretation:
Fig. 3. Graphical representation of the sine function
A distinctive feature of this mode is the minimal consumption of resources. The read-write operations on the hard drive are minimized, tester agents do not download quotes of the requested symbols, no additional computations, all calculations are focused in the OnTester function.
Given the high-speed potential of OnTester, it is possible to create a self-sufficient computation module able to perform simple simulations. Here are the elements of this module:
■History of the symbol for testing
■Virtual position system
■Trading system for managing virtual positions
■Result analysis system
The self-sufficiency of the module means that a single expert will contain all the necessary data for testing and the testing system itself, which will use them. This expert can be easily passed to a distributed computing network in case a cloud optimization is required.
Let us move on to description of the first part of the system, namely, how to store the history for testing.
Saving the symbol history data for the tester based on mathematical calculations
The math calculations mode does not imply access to trading instruments. Calling functions like CopyRates(Symbol(),...) is meaningless here. However, historical data are necessary for the simulation. For this purpose, the quotes history of the required symbol can be stored in a precompressed array of the uchar[] type:
uchar symbol[128394] = {0x98,0x32,0xa6,0xf7,0x64,0xbc...};Any type of data — sound, image, numbers and strings — can be represented as a simple set of bytes. A byte is a short block consisting of eight bits. Any information is stored as "batches" in a sequence consisting of these bytes. MQL5 has a special data type — uchar, where each value can represent exactly one byte. Thus, an array of uchar type with 100 elements can store 100 bytes.
Quotes of a symbol consist of many bars. Each bar includes the information about the bar opening time, its prices (High, Low, Open and Close) and volume. Each such value is stored in a variable of the appropriate length. Here is the table:
Value Data type Size in bytes
Open Time datetime 8
Open double 8
High double 8
Low double 8
Close double 8
Tick volume long 8
Spread int 4
Real volume long 8
It is easy to calculate that storing one bar will require 60 bytes, or a uchar array consisting of 60 elements. For a 24-hour Forex market, one trading day consists of 1,440 minute bars. Consequently, one-minute history of one year consists of approximately 391,680 bars. Multiplying this number by 60 bytes, we find out that one year of a minute history in uncompressed form takes up approximately 23 MB. Is this a lot or a little? It is not much by modern standards, but imagine what happens if we decide to test an expert on the data for 10 years. It will be necessary to store 230 MB of data, and maybe even distribute them over a network. This is very much even by modern standards.
Therefore, it is necessary to somehow compress this information. Fortunately, a special library has been written for working with Zip archives. In addition to various features, this library allows converting the compression result into an array of bytes, which greatly facilitates out work.
So, our algorithm will load the MqlRates array of bars, convert it into a byte representation, then compress it with the Zip archiver, and save the compressed data as a uchar array defined in the mqh file.
To convert quotes into a byte array, the system of conversion via the union type will be used. This system allows placing several data types in one storage field. Thus, it is possible to access the data of one type by addressing another. Such a union will store two types: the MqlRates structure and the uchar array, with the number of elements equal to the size of MqlRates. To understand how this system works, refer to the first version of the SaveRates.mq5 script, which converts the symbol history into a uchar byte array:
//+------------------------------------------------------------------+
//| SaveRates.mq5 |
//| Copyright 2016, Vasiliy Sokolov. |
//| http://www.mql5.com |
//+------------------------------------------------------------------+
#property copyright "Copyright 2016, Vasiliy Sokolov."
#property link "http://www.mql5.com"
#property version "1.00"
#include <Zip\Zip.mqh>
#include <ResourceCreator.mqh>
input ENUM_TIMEFRAMES MainPeriod;
union URateToByte
{
MqlRates bar;
uchar bar_array[sizeof(MqlRates)];
}RateToByte;
//+------------------------------------------------------------------+
//| Script program start function |
//+------------------------------------------------------------------+
void OnStart()
{
//-- Download quotes
MqlRates rates[];
int total = CopyRates(Symbol(), Period(), 0, 20000, rates);
uchar symbol_array[];
//-- Convert them to a byte representation
ArrayResize(symbol_array, sizeof(MqlRates)total);
for(int i = 0, dst = 0; i < total; i++, dst +=sizeof(MqlRates))
{
RateToByte.bar = rates[i];
ArrayCopy(symbol_array, RateToByte.bar_array, dst, 0, WHOLE_ARRAY);
}
//-- Compress them into a zip archive
CZip Zip;
CZipFile file = new CZipFile(Symbol(), symbol_array);
Zip.AddFile(file);
uchar zip_symbol[];
//-- Get the byte representation of the compressed archive
Zip.ToCharArray(zip_symbol);
//-- Write it as a mqh include file
CCreator creator;
creator.ByteArrayToMqhArray(zip_symbol, "rates.mqh", "rates");
}
//+------------------------------------------------------------------+
After this code is executed, the zip_symbol array will contain a compressed array of MqlRates structures — compressed history of quotes. Then the compressed array is stored as a mqh file on the computer hard drive. Details on how and why this is done are provided below.
Obtaining a byte representation of quotes and compressing them is not sufficient. It is necessary to record this representation as a uchar array. In this case, the array should be loaded in the form of a resource, i.e. it must be compiled with the program. For this purpose, create a special mqh header file containing this array as a simple set of ASCII characters. To do this, use the special CResourceCreator class:
//+------------------------------------------------------------------+
//| ResourceCreator.mqh |
//| Copyright 2017, Vasiliy Sokolov. |
//| http://www.mql5.com |
//+------------------------------------------------------------------+
#property copyright "Copyright 2017, Vasiliy Sokolov."
#property link "http://www.mql5.com"
#include <Arrays\ArrayObj.mqh>
//+------------------------------------------------------------------+
//| Contains the string identifiers of the created resource array |
//+------------------------------------------------------------------+
class CResInfo : public CObject
{
public:
string FileName;
string MqhFileName;
string ArrayName;
};
//+------------------------------------------------------------------+
//| Creates a MQL resource as a byte array. |
//+------------------------------------------------------------------+
class CCreator
{
private:
int m_common;
bool m_ch[256];
string ToMqhName(string name);
void CreateInclude(CArrayObj* list_info, string file_name);
public:
CCreator(void);
void SetCommonDirectory(bool common);
bool FileToByteArray(string file_name, uchar& byte_array[]);
bool ByteArrayToMqhArray(uchar& byte_array[], string file_name, string array_name);
void DirectoryToMqhArray(string src_dir, string dst_dir, bool create_include = false);
};
//+------------------------------------------------------------------+
//| Default constructor |
//+------------------------------------------------------------------+
CCreator::CCreator(void) : m_common(FILE_COMMON)
{
ArrayInitialize(m_ch, false);
for(uchar i = '0'; i < '9'; i++)
m_ch[i] = true;
for(uchar i = 'A'; i < 'Z'; i++)
m_ch[i] = true;
}
//+------------------------------------------------------------------+
//| Sets the FILE_COMMON flag, or removes it |
//+------------------------------------------------------------------+
CCreator::SetCommonDirectory(bool common)
{
m_common = common ? FILE_COMMON : 0;
}
//+------------------------------------------------------------------+
//| Converts all files in the src_dir directory to mqh files |
//| containing the byte representation of these files |
//+------------------------------------------------------------------+
void CCreator::DirectoryToMqhArray(string src_dir,string dst_dir, bool create_include = false)
{
string file_name;
string file_mqh;
CArrayObj list_info;
long h = FileFindFirst(src_dir+"\", file_name, m_common);
if(h == INVALID_HANDLE)
{
printf("Directory" + src_dir + " is not found, or it does not contain files");
return;
}
do
{
uchar array[];
if(FileToByteArray(src_dir+file_name, array))
{
string norm_name = ToMqhName(file_name);
file_mqh = dst_dir + norm_name + ".mqh";
ByteArrayToMqhArray(array, file_mqh, "m_"+norm_name);
printf("Create resource: " + file_mqh);
// Add information about the created resource
CResInfo info = new CResInfo();
list_info.Add(info);
info.FileName = file_name;
info.MqhFileName = norm_name + ".mqh";
info.ArrayName = "m_"+norm_name;
}
}while(FileFindNext(h, file_name));
if(create_include)
CreateInclude(&list_info, dst_dir+"include.mqh");
}
//+------------------------------------------------------------------+
/| Creates a mqh file with inclusions of all generated files |
//+------------------------------------------------------------------+
void CCreator::CreateInclude(CArrayObj list_info, string file_name)
{
int handle = FileOpen(file_name, FILE_WRITE|FILE_TXT|m_common);
if(handle == INVALID_HANDLE)
{
printf("Failed to create the include file " + file_name);
return;
}
// Create the include header
for(int i = 0; i < list_info.Total(); i++)
{
CResInfo info = list_info.At(i);
string line = "#include "" + info.MqhFileName + ""\n";
FileWriteString(handle, line);
}
// Create a function for copying the resource array to the calling code
FileWriteString(handle, "\n");
FileWriteString(handle, "void CopyResource(string file_name, uchar &array[])\n");
FileWriteString(handle, "{\n");
for(int i = 0; i < list_info.Total(); i++)
{
CResInfo* info = list_info.At(i);
if(i == 0)
FileWriteString(handle, " if(file_name == "" + info.FileName + "")\n");
else
FileWriteString(handle, " else if(file_name == "" + info.FileName + "")\n");
FileWriteString(handle, " ArrayCopy(array, " + info.ArrayName + ");\n");
}
FileWriteString(handle, "}\n");
FileClose(handle);
}
//+------------------------------------------------------------------+
//| converts the passed name into a correct name of the MQL variable |
//+------------------------------------------------------------------+
string CCreator::ToMqhName(string name)
{
uchar in_array[];
uchar out_array[];
int total = StringToCharArray(name, in_array);
ArrayResize(out_array, total);
int t = 0;
for(int i = 0; i < total; i++)
{
uchar ch = in_array[i];
if(m_ch[ch])
out_array[t++] = ch;
else if(ch == ' ')
out_array[t++] = '_';
uchar d = out_array[t-1];
int dbg = 4;
}
string line = CharArrayToString(out_array, 0, t);
return line;
}
//+------------------------------------------------------------------+
//| Returns the byte representation of the passed file as the |
//| byte_array array |
//+------------------------------------------------------------------+
bool CCreator::FileToByteArray(string file_name, uchar& byte_array[])
{
int handle = FileOpen(file_name, FILE_READ|FILE_BIN|m_common);
if(handle == -1)
{
printf("Failed to open file " + file_name + ". Reason: " + (string)GetLastError());
return false;
}
FileReadArray(handle, byte_array, WHOLE_ARRAY);
FileClose(handle);
return true;
}
//+------------------------------------------------------------------+
//| Converts the passed byte_array byte array into a mqh file named |
//| file_name which contains the description of an array named |
//| array_name |
//+------------------------------------------------------------------+
bool CCreator::ByteArrayToMqhArray(uchar& byte_array[], string file_name, string array_name)
{
int size = ArraySize(byte_array);
if(size == 0)
return false;
int handle = FileOpen(file_name, FILE_WRITE|FILE_TXT|m_common, "");
if(handle == -1)
return false;
string strSize = (string)size;
string strArray = "uchar " +array_name + "[" + strSize + "] = \n{\n";
FileWriteString(handle, strArray);
string line = " ";
int chaptersLine = 32;
for(int i = 0; i < size; i++)
{
ushort ch = byte_array[i];
line += (string)ch;
if(i == size - 1)
line += "\n";
if(i>0 && i%chaptersLine == 0)
{
if(i < size-1)
line += ",\n";
FileWriteString(handle, line);
line = " ";
}
else if(i < size - 1)
line += ",";
}
if(line != "")
FileWriteString(handle, line);
FileWriteString(handle, "};");
FileClose(handle);
return true;
}
Let us not dwell on its operation in detail, and only describe it in general terms, listing its features.
■Reads any arbitrary file on the hard drive and stores its byte representation as a uchar array in a mqh file.
■Reads any arbitrary directory on the hart drive and stores the byte representation of all files located in this directory. The byte representation for each such file is located in a separate mqh file containing a uchar array.
■It takes the uchar array of bytes as input and stores it as an array of characters in a separate mqh file.
■Creates a special header file that contains links to all mqh files created during the generation process. Also, a special function is created, which takes the array name as input and returns its byte representation. This algorithm uses dynamic code generation.
The described class is a powerful alternative to the regular resource allocation system in a MQL program.
By default, all file operations take place in the shared file directory (FILE_COMMON). If you run the script from the previous listing, the folder will have a new rates.mqh file (the file name is defined by the second parameter of the ByteArrayToMqhArray method). It will contain a giant rates[] array (the array name is defined by the third parameter of this method). Here is a snippet of the file contents:
Fig. 4. MqlRates quotes as a compressed byte array named rates
Data compression works fine. One year of uncompressed one-minute history for the EURUSD currency pair takes about 20 MB, after compression — only 5 MB. However, it is better not to open the rates.mqh file in MetaEditor: its size is much larger than this figure, and the editor may freeze. But do not worry. After compilation, the text is converted into bytes and the actual size of the program increases only by the real value of the stored information, in this case, by 5 MB.
By the way, this technique can be used in an ex5 program to store necessary information of any type, not just the quotes history.