MiKTeX and proTeXt ⇒ TeX capacity exceeded, sorry [save_size = 50000]

[JethroBodeine]

I've written a system of TeX macros that work recursively to generate certain sets of sequences of integers with prescribed sums. The system works recursively on the length L of the sequence of integers. Starting with a given L, the recursion digs down to L = 1 and then works back up, L = 2,
and so on, finally using length L-1 sequences to generate the length L sequence.
The system generates the sets one at a time, and and writes a definition of the set to an auxiliary file. The idea behind the auxiliary file is to recursively generate the sets just one time and afterwards, to access the definitions of the auxiliary file when the sets are needed. At each step the L-1 sets are undefined (set to \relax) once the L sets are available for the next step. This use of an auxiliary file avoids multiple regenerations, which are slow and, even with undefines, which place several big sets in memory all at once.

I get the error message TeX capacity exceeded, sorry [save_size = 50000] when I try to increase L to 6 or so. The TeXbook tells us that this error is often caused by making both local and global assignments to the same variable. This is indeed the case, I used both \edef and \xdef to make certain assignments. I tried tweaking the code, changing my localization scheme to eliminate mixed assignments to one variable, but so far something always goes wrong, undefined messages or the results are incomplete.

I found posts that suggest that TeX's capacity can be increased. I'm running MikTeX2.9 and I found the file, texmf-defaults.ini, containing the line save_size=50000. I tried editing this file but, as its name suggest, this is not allowed. Is there an editable file in the MikTeX2.9 system holding the current (versus default) value of save_size? The post says also to recompile TeX to regenerate all format files. How is this done?

For now I'm trying to learn something by running TeX with \tracingrestores=1.

I'd be grateful for any help or suggestions. Regards...

[Johannes_B]

A first start can be Ulrikes advice on TeX.SX. Ulrike said there is no need (she knows what she is talking about) to run initexmf. If it should be necessary, you can read about it in the part of the MikTeX manual.

LuaTeX in the other hand allocates memory at run time and will not run into this problem. There is a downside, tough, it is slow (i heard of big improvements in the new version, but couldn't test it yet).

The part where i would start is the recursion itself. You are talking about it, but i don't know what you are doing or trying to do. But i should say, i am not an expert off the recursion art.

[JethroBodeine]

Johannes, thanks, I appreciate your interest. I tried LuaTeX, no help, the same capacity error.

I took the words, "recompile TeX to regenerate all format files," from another post. Now I know what they mean, to run initexmf. Thanks. I have not yet learned how to reset the save_size variable in MikTeX2.9, but I when I do, I'll try the system without recompiling TeX.

[JethroBodeine]

The post I refer to above is the April 4, 2012 post listed below.

[JethroBodeine]

The post I refer to above is the April 4, 2012 post listed below. It explains how to reset save_size in a TeX Live system. I would like to try something like this on my MikTeX system. Please let me know the name of the MikTeX file to edit the current value of save_size.

[Johannes_B]

I already linked Ulrikes answer, you can find more at TeX.SX.

This might be something you were referring to in your first post.
What yuo wanted to say in your last 3 posts (btw, you can edit your own posts) i really don't understand.

[JethroBodeine]

Johannes, Thanks for staying with me on this. I did not immediately translate your Ulrike reference into something I could use. I ran tex with the option --save-size = n where n = 60000, 70000, 80000, 90000 (50000 is the default).
I got the capacity error TeX capacity exceeded, sorry [save_size = n] in the first 3 tries and finally with n = 90000 I was told that this is a bad parameter value. I also checked the MikTeX manual on the program tex. It mentions the save-size option but does not provide its limits. This looks like a dead-end to me, my recursion requires more save-size than what is allowed.

I will continue to try to understand the output of a tex run with \tracingrestores=1. The trace output contains two statements,
{restoring \var ...}
and
{retaining \var ...}
and, according to the TeXbook page 301, "If you find TEX retaining a lot of values, you have a set of macros that could cause the save stack to overflow..." The log file of a successful run with L set to a modest 4 contains over 31,000 restoring/retaining entries. What follows is just a small portion of that log file. My problem now is one of following the trace and assigning a part of the recursion to a line or portion of the output of the trace.
{restoring \ifeqcsb=macro:->2}
{restoring \ifeqcsa=macro:->9}
{retaining \ifeqcsb=macro:->2}
{retaining \ifeqcsa=macro:->9}
{retaining \ifeqcsb=macro:->2}
{retaining \ifeqcsa=macro:->9}
{retaining \ifeqcsb=macro:->2}
{retaining \ifeqcsa=macro:->9}
{retaining \ifeqcsb=macro:->2}
{retaining \ifeqcsa=macro:->9}
{retaining \ifeqcsb=macro:->2}
{retaining \ifeqcsa=macro:->9}
{retaining \ifeqcsb=macro:->2}
{retaining \ifeqcsa=macro:->9}
{retaining \ifeqcsb=macro:->2}
{retaining \ifeqcsa=macro:->9}
{retaining \ifeqcsb=macro:->2}
{retaining \ifeqcsa=macro:->9}
{retaining \ifeqcsb=macro:->2}
{retaining \ifeqcsa=macro:->9}
{restoring \nxt=undefined}

[Johannes_B]

Your are trying to make the result fit within some limits by forcing the limits to increase.
You should take care, that your result cannot touch the limit.

Lot's of use have run out of memory, sure, adding to the memory helped. But you are talking of recursion, writing to files and stuff like that. Recursion can be fun, but it can be hell as well. If you want to understand the line-breaking mechanism, three or four paragraphs of output should be enough to see what's going on. Are you trying to improve it by setting penalties? Next time, try some other paragraphs. Compare.


As long as i don't understand the problem, i can't give any reasonable hint (tough i never really looked into line breaking).

[JethroBodeine]

I found one of my utility macros, which is used often in the recursion, with two variables unnecessarily defined using \xdef. I changed these to \edef, as suggested by p. 301 of the \TeXbook, and the program ran for values of the parameter L up to 6 or 7. Then I increased L to 9 and got another, but different capacity error, TeX capacity exceeded, sorry [main memory size=3000000]. I googled this problem and found a post, http://tex.stackexchange.com/questions/ ... ze-3000000, that suggested LuaTeX, because it allocates memory dynamically, might help here. With the correction to my code to avoid overflowing the save stack and the use of LuaTeX to expand memory, it appears that I'm on my way.

I'll explain briefly what this is all about. It's not really about typesetting documents, more about using TeX's ability to manipulate data to do mathematics. I'm applying statistical techniques of pattern recognition, including category theory and latency, to problems in gambling. One of these, BlackJack, uses sets S(L,s), the set of all hands of length L (mentioned frequently above) whose cards sum to s. These sets satisfy a difference equation which provides a new approach to measuring the probabilities of this game. The system of macros I talked about previously take a recursive approach to solving the equation. An example of a very similar difference equation is the well known Fibonacci equation, S_n = S_{n-1} + S{n-2}. Starting with S_0 = 1 and S_1 = 1, the equation gives S_2 = 2, S_3 = 3, S_4 = 5, S_5 = 8, and so on. You can see, since S_n depends on S_{n-1} and S{n-2}, it is natural to use recursion with respect to n to generate this sequence. The S(L,s), however, are sets, not numbers. I wrote a family of TeX operators/macros to manipulate sets, including \ifequal, \ifelement, \ifcontains, \ifnumerical, \intersection, \remove, \union, \product, much as programmers define methods to manipulate the objects of a class. These macros are used in the recursion to generate the S(L,s) and the first of these, \ifequal, is the culprit that used \xdef unnecessarily.

[Johannes_B]

TeX's ability to manipulate data to do mathematics.

I stopped right there reading.

TeX is ment for typesetting, it can do basic maths, but ...

Use a proper tool for every task. You can lumber trees using a spoon, might take a while tough. But a proper tool would be an axe or even a chainsaw (if you have to do it multiple times).