Merge branch 'patch-remove-py2'

Author: defnull

bottle.py

@@ -36,14 +36,19 @@ to receive updates on a best-effort basis.
 Release 0.14 (in development)
 =============================
 
-.. rubric:: Removed APIs (deprecated since 0.13)
+.. rubric:: Removed APIs
 
+* Dropped support for Python 2 and removed workarounds or helpers that only make sense in a Python 2/3 dual codebase.
 * Removed the ``RouteReset`` exception and associated logic.
 * Removed the `bottle.py` console script entrypoint in favour of the new `bottle` script. You can still execute `bottle.py` directly or via `python -m bottle`. The only change is that the command installed by pip or similar tools into the bin/Scripts folder of the (virtual) environment is now called `bottle` to avoid circular import errors.
 
 .. rubric:: Changes
 
-* ``bottle.HTTPError`` raised on Invalid JSON now include the underlying exception in their ``exception`` field.
+* Form values, query parameters, path elements and cookies are now always decoded as `utf8` with `errors='surrogateescape'`. This is the correct approach for almost all modern web applications, but still allows applications to recover the original byte sequence if needed. This also means that ``bottle.FormsDict`` no longer re-encodes PEP-3333 `latin1` strings to `utf8` on demand (via attribute access). The ``FormsDict.getunicode()`` and ``FormsDict.decode()`` methods are deprecated and do nothing, as all values are already transcoded to `utf8`.
+
+.. rubric:: New features
+
+* ``bottle.HTTPError`` raised on Invalid JSON now include the underlying exception in the ``exception`` field.
 
 
 Release 0.13
@@ -74,7 +79,7 @@ versions should not update to Bottle 0.13 and stick with 0.12 instead.
 
 .. rubric:: Deprecated APIs
 
-* Python 2 support is now deprecated and will be dropped with the next release.
+* Python 2 support is now deprecated and will be dropped with the next release. This includes helpers and workarounds that only make sense in a Python 2/3 dual codebase (e.g. ``tonat()`` or the ``py3k`` flag). 
 * The command line executable installed along with bottle will be renamed from `bottle.py` to just `bottle`. You can still execute bottle directly as a script (e.g. `./bottle.py` or `python3 bottle.py`) or as a module (via `python3 -m bottle`). Just the executable installed by your packaging tool (e.g. `pip`) into the `bin` folder of your (virtual) environment will change.
 * The old route syntax (``/hello/:name``) is deprecated in favor of the more readable and flexible ``/hello/<name>`` syntax.
 * :meth:`Bottle.mount` now recognizes Bottle instance and will warn about parameters that are not compatible with the new mounting behavior. The old behavior (mount applications as WSGI callable) still works and is used as a fallback automatically.

docs/changelog.rst

@@ -552,28 +552,18 @@ Property                                       Data source
 
 Bottle uses a special type of dictionary to store those parameters. :class:`FormsDict` behaves like a normal dictionary, but has some additional features to make your life easier.
 
-First of all, :class:`FormsDict` is a subclass of :class:`MultiDict` and can store more than one value per key. The standard dictionary access methods will only return the first of many values, but the :meth:`MultiDict.getall` method returns a (possibly empty) list of all values for a specific key::
+First of all, :class:`FormsDict` is a subclass of :class:`MultiDict` and can store more than one value per key. Only the first value is returned by default, but :meth:`MultiDict.getall` can be used to get a (possibly empty) list of all values for a specific key::
 
   for choice in request.forms.getall('multiple_choice'):
       do_something(choice)
 
-To simplify dealing with lots of unreliable user input, :class:`FormsDict` exposes all its values as attributes, but with a twist: These virtual attributes always return properly encoded unicode strings, even if the value is missing or character decoding fails. They never return ``None`` or throw an exception, but return an empty string instead::
+Attribute-like access is also supported, returning empty strings for missing values. This simplifies code a lot whend ealing with lots of optional attributes::
 
   name = request.query.name    # may be an empty string
 
 .. rubric:: A word on unicode and character encodings
 
-HTTP is a byte-based wire protocol. The server has to decode byte strings somehow before they are passed to the application. To be on the safe side, WSGI suggests ISO-8859-1 (aka latin1), a reversible single-byte codec that can be re-encoded with a different encoding later. Bottle does that for :meth:`FormsDict.getunicode` and attribute access, but not for :meth:`FormsDict.get` or item-access. These return the unchanged values as provided by the server implementation, which is probably not what you want.
-
-::
-
-    >>> request.query['city']
-    'Göttingen' # An utf8 string provisionally decoded as ISO-8859-1 by the server
-    >>> request.query.city
-    'Göttingen'  # The same string correctly re-encoded as utf8 by bottle
-
-If you need the whole dictionary with correctly decoded values (e.g. for WTForms), you can call :meth:`FormsDict.decode` to get a fully re-encoded copy.
-
+Unicode characters in the request path, query parameters or cookies are a bit tricky. HTTP is a very old byte-based protocol that predates unicode and lacks explicit encoding information. This is why WSGI servers have to fall back on `ISO-8859-1` (aka `latin1`, a reversible input encoding) for those estrings. Modern browsers default to `utf8`, though. It's a bit much to ask application developers to translate every single user input string to the correct encoding manually. Bottle makes this easy and just assumes `utf8` for everything. All strings returned by Bottle APIs support the full range of unicode characters, as long as the webpage or HTTP client follows best practices and does not break with established standards.
 
 Query Parameters
 --------------------------------------------------------------------------------

docs/tutorial.rst

@@ -7,7 +7,7 @@
 import itertools
 
 import bottle
-from bottle import request, tob, touni, tonat, json_dumps, HTTPError, parse_date, CookieError
+from bottle import request, tob, touni, json_dumps, HTTPError, parse_date, CookieError
 from . import tools
 import wsgiref.util
 import base64
@@ -160,16 +160,16 @@ def test_cookie_dict(self):
 
     def test_get(self):
         """ Environ: GET data """
-        qs = tonat(tob('a=a&a=1&b=b&c=c&cn=%e7%93%b6'), 'latin1')
+        qs = touni(tob('a=a&a=1&b=b&c=c&cn=%e7%93%b6'), 'latin1')
         request = BaseRequest({'QUERY_STRING':qs})
         self.assertTrue('a' in request.query)
         self.assertTrue('b' in request.query)
         self.assertEqual(['a','1'], request.query.getall('a'))
         self.assertEqual(['b'], request.query.getall('b'))
         self.assertEqual('1', request.query['a'])
         self.assertEqual('b', request.query['b'])
-        self.assertEqual(tonat(tob('瓶'), 'latin1'), request.query['cn'])
-        self.assertEqual(touni('瓶'), request.query.cn)
+        self.assertEqual('瓶', request.query['cn'])
+        self.assertEqual('瓶', request.query.cn)
 
     def test_post(self):
         """ Environ: POST data """
@@ -189,8 +189,8 @@ def test_post(self):
         self.assertEqual('b', request.POST['b'])
         self.assertEqual('', request.POST['c'])
         self.assertEqual('', request.POST['d'])
-        self.assertEqual(tonat(tob('瓶'), 'latin1'), request.POST['cn'])
-        self.assertEqual(touni('瓶'), request.POST.cn)
+        self.assertEqual('瓶', request.POST['cn'])
+        self.assertEqual('瓶', request.POST.cn)
 
     def test_bodypost(self):
         sq = tob('foobar')
@@ -503,15 +503,11 @@ def cmp(app, wire):
             result = [v for (h, v) in rs.headerlist if h.lower()=='x-test'][0]
             self.assertEqual(wire, result)
 
-        if bottle.py3k:
-            cmp(1, tonat('1', 'latin1'))
-            cmp('öäü', 'öäü'.encode('utf8').decode('latin1'))
-            # Dropped byte header support in Python 3:
-            #cmp(tob('äöü'), 'äöü'.encode('utf8').decode('latin1'))
-        else:
-            cmp(1, '1')
-            cmp('öäü', 'öäü')
-            cmp(touni('äöü'), 'äöü')
+        cmp(1, touni('1', 'latin1'))
+        cmp('öäü', 'öäü'.encode('utf8').decode('latin1'))
+        # Dropped byte header support in Python 3:
+        #cmp(tob('äöü'), 'äöü'.encode('utf8').decode('latin1'))
+
 
     def test_set_status(self):
         rs = BaseResponse()
@@ -583,12 +579,11 @@ def test(): rs.status = '555' # No reason
         self.assertEqual(rs.status_line, '404 Brain not Found') # last value
 
         # Unicode in status line (thanks RFC7230 :/)
-        if bottle.py3k:
-            rs.status = '400 Non-ASÎÎ'
-            self.assertEqual(rs.status, rs.status_line)
-            self.assertEqual(rs.status_code, 400)
-            wire = rs._wsgi_status_line().encode('latin1')
-            self.assertEqual(rs.status, wire.decode('utf8'))
+        rs.status = '400 Non-ASÎÎ'
+        self.assertEqual(rs.status, rs.status_line)
+        self.assertEqual(rs.status_code, 400)
+        wire = rs._wsgi_status_line().encode('latin1')
+        self.assertEqual(rs.status, wire.decode('utf8'))
 
     def test_content_type(self):
         rs = BaseResponse()
@@ -735,7 +730,7 @@ def test_non_string_header(self):
         response['x-test'] = None
         self.assertEqual('', response['x-test'])
         response['x-test'] = touni('瓶')
-        self.assertEqual(tonat(touni('瓶')), response['x-test'])
+        self.assertEqual(touni('瓶'), response['x-test'])
 
     def test_prevent_control_characters_in_headers(self):
         masks = '{}test', 'test{}', 'te{}st'
@@ -895,10 +890,6 @@ def test_native(self):
         self.env['HTTP_TEST_HEADER'] = 'foobar'
         self.assertEqual(self.headers['Test-header'], 'foobar')
 
-    def test_bytes(self):
-        self.env['HTTP_TEST_HEADER'] = tob('foobar')
-        self.assertEqual(self.headers['Test-Header'], 'foobar')
-
     def test_unicode(self):
         self.env['HTTP_TEST_HEADER'] = touni('foobar')
         self.assertEqual(self.headers['Test-Header'], 'foobar')

test/test_environ.py

@@ -33,8 +33,7 @@ def test_filename(self):
         self.assertFilename('.name.cfg', 'name.cfg')
         self.assertFilename(' . na me . ', 'na-me')
         self.assertFilename('path/', 'empty')
-        self.assertFilename(bottle.tob('ümläüts$'), 'umlauts')
-        self.assertFilename(bottle.touni('ümläüts$'), 'umlauts')
+        self.assertFilename('ümläüts$', 'umlauts')
         self.assertFilename('', 'empty')
         self.assertFilename('a'+'b'*1337+'c', 'a'+'b'*254)
 

test/test_fileupload.py

@@ -7,25 +7,11 @@
 class TestFormsDict(unittest.TestCase):
     def test_attr_access(self):
         """ FomsDict.attribute returs string values as unicode. """
-        d = FormsDict(py2=tob('瓶'), py3=tob('瓶').decode('latin1'))
-        self.assertEqual(touni('瓶'), d.py2)
-        self.assertEqual(touni('瓶'), d.py3)
+        d = FormsDict(py3='瓶')
+        self.assertEqual('瓶', d.py3)
+        self.assertEqual('瓶', d["py3"])
 
     def test_attr_missing(self):
         """ FomsDict.attribute returs u'' on missing keys. """
         d = FormsDict()
-        self.assertEqual(touni(''), d.missing)
-
-    def test_attr_unicode_error(self):
-        """ FomsDict.attribute returs u'' on UnicodeError. """
-        d = FormsDict(latin=touni('öäüß').encode('latin1'))
-        self.assertEqual(touni(''), d.latin)
-        d.input_encoding = 'latin1'
-        self.assertEqual(touni('öäüß'), d.latin)
-
-    def test_decode_method(self):
-        d = FormsDict(py2=tob('瓶'), py3=tob('瓶').decode('latin1'))
-        d = d.decode()
-        self.assertFalse(d.recode_unicode)
-        self.assertTrue(hasattr(list(d.keys())[0], 'encode'))
-        self.assertTrue(hasattr(list(d.values())[0], 'encode'))
+        self.assertEqual('', d.missing)

test/test_formsdict.py

@@ -59,9 +59,8 @@ def x(a, b):
         # triggers the "TypeError: 'foo' is not a Python function"
         self.assertEqual(set(route.get_callback_args()), set(['a', 'b']))
 
-    if bottle.py3k:
-        def test_callback_inspection_newsig(self):
-            env = {}
-            eval(compile('def foo(a, *, b=5): pass', '<foo>', 'exec'), env, env)
-            route = bottle.Route(bottle.Bottle(), None, None, env['foo'])
-            self.assertEqual(set(route.get_callback_args()), set(['a', 'b']))
+    def test_callback_inspection_newsig(self):
+        env = {}
+        eval(compile('def foo(a, *, b=5): pass', '<foo>', 'exec'), env, env)
+        route = bottle.Route(bottle.Bottle(), None, None, env['foo'])
+        self.assertEqual(set(route.get_callback_args()), set(['a', 'b']))

test/test_route.py

@@ -102,7 +102,7 @@ def test_ims(self):
         res = static_file(basename, root=root)
         self.assertEqual(304, res.status_code)
         self.assertEqual(int(os.stat(__file__).st_mtime), parse_date(res.headers['Last-Modified']))
-        self.assertAlmostEqual(int(time.time()), parse_date(res.headers['Date']))
+        self.assertAlmostEqual(int(time.time()), parse_date(res.headers['Date']), delta=2)
         request.environ['HTTP_IF_MODIFIED_SINCE'] = bottle.http_date(100)
         self.assertEqual(open(__file__,'rb').read(), static_file(basename, root=root).body.read())
 

test/test_sendfile.py

@@ -103,9 +103,7 @@ def test(string): return string
         self.assertBody(tob('urf8-öäü'), '/my-öäü/urf8-öäü')
 
     def test_utf8_header(self):
-        header = 'öäü'
-        if bottle.py3k:
-            header = header.encode('utf8').decode('latin1')
+        header = 'öäü'.encode('utf8').decode('latin1')
         @bottle.route('/test')
         def test():
             h = bottle.request.get_header('X-Test')

test/test_wsgi.py

@@ -13,7 +13,7 @@
 import mimetypes
 import uuid
 
-from bottle import tob, tonat, BytesIO, py3k, unicode
+from bottle import tob, BytesIO
 
 
 def warn(msg):
@@ -76,10 +76,7 @@ def decorator(func):
 
 
 def wsgistr(s):
-    if py3k:
-        return s.encode('utf8').decode('latin1')
-    else:
-        return s
+    return s.encode('utf8').decode('latin1')
 
 class ServerTestBase(unittest.TestCase):
     def setUp(self):
@@ -170,7 +167,7 @@ def multipart_environ(fields, files):
         body += 'Content-Type: %s\r\n\r\n' % mimetype
         body += content + '\r\n'
     body += boundary + '--\r\n'
-    if isinstance(body, unicode):
+    if isinstance(body, str):
         body = body.encode('utf8')
     env['CONTENT_LENGTH'] = str(len(body))
     env['wsgi.input'].write(body)