UnicodeDecodeError issue with downloaded .html file

[ghost]

Hi, first of all thanks for this repository. I've tried to use pywebcopy but as it was mentioned here it hangs at some point and can't even interrupt. pywebcopy7 seems much better and faster than previous, however I keep getting UnicodeDecodeError when I try to open the HTML file with Flask. I tried to convert it to utf-8 with Notepad++ but it didn't change at all.

It works like charm for very small websites, such as http://example.com/ .
But when I try for more complicated websites, like news web sites, the .html file seems like this in the browser:

When I locally try to create a web page by using the html file, with Flask, it gives this error:

* Running on http://127.0.0.1:5000/ (Press CTRL+C to quit) [2020-10-07 11:40:13,621] ERROR in app: Exception on / [GET] Traceback (most recent call last): File "c:\users\test\appdata\local\programs\python\python38\lib\site-packages\flask\app.py", line 2447, in wsgi_app response = self.full_dispatch_request() File "c:\users\test\appdata\local\programs\python\python38\lib\site-packages\flask\app.py", line 1952, in full_dispatch_request rv = self.handle_user_exception(e) File "c:\users\test\appdata\local\programs\python\python38\lib\site-packages\flask\app.py", line 1821, in handle_user_exception reraise(exc_type, exc_value, tb) File "c:\users\test\appdata\local\programs\python\python38\lib\site-packages\flask\_compat.py", line 39, in reraise raise value File "c:\users\test\appdata\local\programs\python\python38\lib\site-packages\flask\app.py", line 1950, in full_dispatch_request rv = self.dispatch_request() File "c:\users\test\appdata\local\programs\python\python38\lib\site-packages\flask\app.py", line 1936, in dispatch_request return self.view_functions[rule.endpoint](**req.view_args) File "C:\Users\test\Desktop\g_cloner\local_deneme.py", line 17, in home return render_template('milliyet_index.html') File "c:\users\test\appdata\local\programs\python\python38\lib\site-packages\flask\templating.py", line 138, in render_template ctx.app.jinja_env.get_or_select_template(template_name_or_list), File "c:\users\test\appdata\local\programs\python\python38\lib\site-packages\jinja2\environment.py", line 930, in get_or_select_template return self.get_template(template_name_or_list, parent, globals) File "c:\users\test\appdata\local\programs\python\python38\lib\site-packages\jinja2\environment.py", line 883, in get_template return self._load_template(name, self.make_globals(globals)) File "c:\users\test\appdata\local\programs\python\python38\lib\site-packages\jinja2\environment.py", line 857, in _load_template template = self.loader.load(self, name, globals) File "c:\users\test\appdata\local\programs\python\python38\lib\site-packages\jinja2\loaders.py", line 115, in load source, filename, uptodate = self.get_source(environment, name) File "c:\users\test\appdata\local\programs\python\python38\lib\site-packages\flask\templating.py", line 60, in get_source return self._get_source_fast(environment, template) File "c:\users\test\appdata\local\programs\python\python38\lib\site-packages\flask\templating.py", line 86, in _get_source_fast return loader.get_source(environment, template) File "c:\users\test\appdata\local\programs\python\python38\lib\site-packages\jinja2\loaders.py", line 184, in get_source contents = f.read().decode(self.encoding) UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte 127.0.0.1 - - [07/Oct/2020 11:40:13] "GET / HTTP/1.1" 500 -

Is there any possible solution for this error?

[rajatomar788]

@mulaixi

Pywebcopy automatically interprets the response to determine the encoding, but if it is not working properly then you can always set the encoding manually using the config key encoding.

Pywebcopy7 is indeed much better but it breaks sometimes. But if it works for your usecase then you can always use it.

Now as per the log of the flask app you posted, the flask server is showing a status code 500 which translates to http server error meaning your flask app is broken.

[ghost]

@rajatomar788 but I did with previous version (which hangs ), same flask app is working properly. By the way, I solved my problem by using previous version as I wrote here. So, I guess we can close it. If someone else encounters the same problem in future they can reopen.