Normalization of number in scientific notation in pgfmath

[Rmano]

This is probably not a bug, but it is quite a surprising thing (disclaimer: probably it is a case of "don't do that", but just to document it...

When using scientific notation, it would be nice to normalize the mantissa before the final conversion. The example below shows the problem:

\documentclass{article}
\usepackage{pgf}
\begin{document}
\pgfmathparse{500e-6}\pgfmathresult{} vs \pgfmathparse{5e-4}\pgfmathresult
\end{document}

This is doubly surprising if you do that inside a pgfplot environment, because in that case the addplot works (I understand it is because it's using some fpu) and the \pgfmathparse not:

\documentclass[border=10pt]{standalone}
\usepackage{pgfplots}\pgfplotsset{compat=newest}
\begin{document}
    %% use 0.0005 or 0.5e.3 and it works
    \newcommand{\valueB}{500e-6}%
    % \newcommand{\valueB}{0.5e-3}%
    \begin{tikzpicture}[declare function={
        NL(\t)=4 + 0.03*(\t) - \valueB*((\t-50)^2);}
    ]
    \begin{axis}[ xmin=0, xmax=90, domain=0:80, ]
        \addplot[color=red, ultra thick] {NL(x)}; % works always
        \pgfmathsetmacro{\Value}{NL(20)} % fails with \valueB 500e-6
        \node [draw,pin=above:\Value] at (20, \Value) {};
    \end{axis}
    \end{tikzpicture}
\end{document}

[hmenke]

\pgfmathscientific (which is internally called to parse literals in scientific notation) assumes everything with exponent less than -4 to be zero, see

pgf/tex/generic/pgf/math/pgfmathfunctions.misc.code.tex

Lines 163 to 165 in 0624b0f

	\ifnum\c@pgfmath@counta<-4\relax%
	\pgfmath@x0pt\relax%
	\else%

That's interesting, because the smallest representable number of a \dimen is 1sp which is (1/65536)pt which rounds to 0.00002pt = 2e-5pt.

[Rmano]

Quite surprising. I think a nice approach would be to normalize the mantissa before that—like taking x = floor(log10(mantissa)), adding x to the exponent, and dividing the mantissa by 10^x. It will slow everything down, but... maybe you could skip it if the mantissa is between 0.1 and 10.

[hmenke]

It's unfortunately not that easy. I need to find a way to fix the parsing for values outside of the currently invalid range while preserving the exact semantics and results for values within the currently valid range. One possibility is to do \count (integer) arithmetic instead of using \dimen arithmetic. The maximum \count in TeX is $2^{31} - 1$ in contrast to \maxdimen which is only $2^{30} - 1$.

The format of the input is mantissa = sign int '.' frac and exponent = sign int.¹ So instead of calculating factor = pow(10, abs(exponent)) with \pgfmathpow@ we could instead you multiply 10 by itself exponent times in a loop which will only overflow for $10^{10}$. Then we convert both int and frac to counters which is equivalent to multiplying by 65536 and rounding the fractional part. For a negative exponent we then have to divide by factor which we have already calculated as a counter. The result of that calculation is prefixed by the sign of the mantissa and converted back to a dimension by suffixing with sp

x = sign * ((int * 65536 + round(frac * 65536) / factor) sp

This is value safe and preserves the correct rounding for the currently valid range and extends the range down to $10^{-9}$. Unfortunately, this also requires rewriting the positive branch, because this will then overflow silently instead of triggering Dimension too large.

Footnotes

1. The exponent format is currently an unenforced contract. ↩