Wednesday, April 11, 2018

Berlin to Copenhagen by bike.

There's a bike path running from the north of Europe all the way to the south. Well, it's not completely finished yet, but it goes most of the way. It's part of the EuroVelo project, which aims to have a high quality cycling network made of 15 routes all around Europe completed by 2020.

All my clothes and possessions for a seven day trip from Berlin to Copenhagen. I was just going to tie a sack under my seat... but was convinced that perhaps these special bags are a good idea after all.
One part that's done well is the Berlin to Copenhagen leg(I guess not so surprising since they are two cities that are really into supporting bikes).
Scroll down on these photos someone took, if you want a look.
http://www.slowtravelberlin.com/in-photos-berlin-copenhagen-cycle-route/

Except, I'm going to do it at the end of April, not in winter like them. Which is usually the most dry time of year, and with a temperature that's not so hot or cold(usually). Fingers crossed it will be dry and warm. It will probably take 7 days if I can ride fast enough each day (about 100km). Eeek. And this part of the route (like many other parts) has a really nice website http://www.bike-berlin-copenhagen.com/route/brandenburg-stage
Going to go very minimal, and take my fast(ish) road bike. With just three sets of clothes(including the ones I'll be wearing), a netbook, and a toothbrush. It probably would be nice to have a tent, but I'll pass. Instead I'll sleep in the forest. If it rains, I'll just get wet.

Just kidding (or am I?). I'll probably cheat and stay in a hotel some nights rather than sleep in forest. I chose the laptop over the tent, because it's nice to be able to sit by a river in a field with no one around and write some nonsense.

A tiny paper notebook and pencils for drawing will also make it into my bike-pack.

Already I've started training a bit, since I didn't ride my bike for much of the winter. The first 6 hour test ride I did left me sore and tired. But that was going from almost zero riding over the last four months and straight into that. Not the smartest idea to go from nothing into such a long ride... but very enjoyable none the less.

Learning from long practice rides around Berlin. Sitting by a lake.
So what have I learnt so far?
  • Padded pants are a good idea.
  • Water is important.
  • Sunscreen is important even on cloudy days.
  • I am not fit.
  • Riding around is one of life's pleasures.
My next long practice ride is going to be with all of my gear on the bike. I expect to learn more. For example, I'll time how far I can go and at what speed. And I also want to try riding in some heavy rain - to test if my bags are actually waterproof. But unfortunately the weather has been pretty nice recently.

Other than experiments I plan to do, I'm reading about what other people do on their long bike rides. And also talking with people. But it's always hard to learn lessons from others, and the best way to find out what is important to me is to do practice rides.

I think I should be able to ride five hours each day with breaks in between, and still get up to my 100km target. But unsure really. By then the days should be quite long, and the nights with a fair bit of light.
  • Berlin day length: 14:30
  • Copenhagen day light length: 15:30
Spring flowers such as 'Tommies' and 'Snow drops' are popping their heads out for spring. Looking forward to seeing all the different landscapes along the way.
With 14-15 sunlight hour days, there's lots of time for slow riding if wind is in my face, for detours, and for sits under trees drawing. By not using the side bags, and not carrying all that much weight I should be able to go a lot faster than other people I've talked to who have done the ride. Their bikes were pretty bad, and they carried a lot more weight than what I'm going to.

I don't really have a good solution for route planning. I have a route I'd like to go, but I can't really put it on google maps easily, so it's visible on an app. So that's another thing I need to find out about. Luckily I've done lots of map development before... but I hope there's not much coding to do such a simple task, that surely other people have done before.

Also, I'm a bit worried about riding in the rain. On average it rains 8 out of 30 days that time of year. So it's quite likely I'll see rain. Generally, I just don't ride my bike if it rains. But I'll have to I guess, because 'waiting it out' doesn't work if it's raining all day.

Tuesday, April 03, 2018

pygame.Surface.blits

Drawing many things at once is faster than drawing things one thing at a time. So, today I made a function to do this with pygame.

It seems to take between 82%-85% of the time it takes to draw things one at a time. Which considering blit is often the bottleneck in many pygame apps is a pretty nice improvement. It will be landing in the pygame 1.9.4 release.

More info in the blits Pull Request.

Moss box

I've been looking to get more greenery into my studio, and I kind of like moss. It reminds me of when the times in my childhood running around rain forests barefoot.

Luckily, a good friend has a whole bunch of it growing out the back of her apartment. And it's likely to be destroyed soon by workers. Which made me not feel so bad about digging some up.


I scavenged some egg cartons

First I found some egg cartons to use as a base. Underneath the moss is a good few centimeters of dirt.

Unlike most of the wall moss I've seen around the place, this moss is alive. I was sort of horrified to learn that people buy dead preserved moss, and glue gun it into place. But this stuff is alive. Sometimes looking stuff up on Youtube kills the magic. Now I don't think wall moss is anywhere near as cool as I once thought it was. Thanks internet.

Here it is sitting on my standing desk.

Next I found an old wooden box. I lined the bottom of this with some plastic I had about the place. Yuk, I know. However, it should hopefully stop any water leaks.

There's two half egg carton containers I found (to hold 12 eggs) being used to fill the box. With spaces around the edge.

The nice thing about having it in a small box is it's easy to move about the place. So I can put it on top of my desk, or even under the desk where my bare feet are. My next project is probably to grow some green grasses in a box for that grass-on-feet-inside feeling.

Can you see the cardboard carton?
I'm not so sure if it will survive, but I'm hoping it will grow. So far I've just been spraying it with water, and occasionally touching and smelling the moss. Which takes me back to the rain forest a bit.

Friday, March 23, 2018

Investigating pypy frame drop

pypy has spikes in time it takes occasionally when using cpyext, otherwise known as pauses.
This is because it's deallocating lots of CPython objects all in one frame, rather than incrementally. This is likely to be addressed in a future release of pypy.

Mainly it's a problem when creating and deleting lots of CPython objects (like pygame.Rect). Not pure python objects.

To work around it for now,
  • the src/rect.c has been changed to have a free list of python objects, so it doesn't alloc and dealloc them. Instead it maintains it's own list of rect PyObject pointers, and reuses them. This is done in PR #431
  • Use PYPY_GC_NURSERY=1m incminimark environment variables when running pypy.
Below are timings of the default pygame, and default pypy (at time of writing). Then a benchmark showing the improvement with the rect freelist implemented in rect.c inside pygame. Then we also improve things by setting PYPY_GC_NURSERY=1m environment variable to change the behavior of the GC. I'm not aware of an API to do this within pygame itself unfortunately. Finally there are some cpython 3.6 timings for comparison.

hacked up benchmark.

The hacked up benchmark is in the branch pypy-hack-frame-bench of the pygame repo. It's based off examples.testsprite.

Because pypy currently doesn't compile matplot lib you need to run it in two steps. First to do the run (let it run for 20+ seconds). Then to load the data pickle, and show the graph.

pypy examples/testsprite.py -plot -noupdate_rects -width 320 -height 200 100
python examples/testsprite.py -plotpickle 

To run with the GC tweak...

PYPY_GC_NURSERY=1m pypy examples/testsprite.py -plot -noupdate_rects -width 320 -height 200 100
 
 

graph colors

  • Blue is the first 1200 frames
  • orange is the last 1200 frames of the run.
This is useful to see if there are any jit warm up affects happening, or different behaviour over time. (doesn't appear so).

Time per frame on pypy 5.10.0.

For this example we don't see any 'jit warm up' behavior.
screen shot 2018-03-23 at 07 21 05

Time per frame on pypy 5.10.0, rect freelist, standard GC.

screen shot 2018-03-23 at 12 05 42

Time per frame on pypy 5.10.0, rect freelist, PYPY_GC_NURSERY=1m.

screen shot 2018-03-23 at 12 04 42
This final result for pypy 'ok' because we want under 0.0166 seconds used per frame. Improvements inside pypy itself in the future should hope to remove many of these spikes.

Time per frame on python3.6

Here we see both the average time per frame and maximums are smaller on python3.6.
screen shot 2018-03-23 at 12 22 02

Time per frame on python3.6, rect freelist

It seems the freelist for rects on python3.6 has a minimal affect.
screen shot 2018-03-23 at 12 20 12

Time per frame on python3.6, gc.disable()

Here we can see using gc.disable() seems to have no affect in this program (on python 3.6). I also tried this on pypy, and it had no affect either (on this program).
screen shot 2018-03-23 at 12 25 14
Other things I tried was to do a gc.collect(0) just before display.flip(), however this didn't appear to help in this case. This is because we can know if we have 5-10ms free waiting for the display VSYNC to flip (when running at 60Hz). Perhaps this can be attempted again as the cpyext GC is improved.

Thursday, March 22, 2018

Windows pypy pygame build for testing.

How to install pypy (fast python written in python) on windows and use the pygame dev build.

pypy running pygame.examples.aliens on Windows

1) Get pypy. It's just a zip folder with all the stuff inside.
https://bitbucket.org/pypy/pypy/downloads/pypy2-v5.10.0-win32.zip
Unzip, and put into C:\pypy, so C:\pypy\pypy.exe exists.

OR get the pypy3.exe from https://bitbucket.org/pypy/pypy/downloads/pypy3-v5.10.1-win32.zip
Unzip, and put into C:\pypy3, so C:\pypy3\pypy3.exe exists.


2) Set the PATH environment variable, so that pypy.exe is found in your command prompt.
https://www.opentechguides.com/how-to/article/windows-10/113/windows-10-set-path.html

Or you can just do this each time you run a new cmd prompt.
set PATH=%PATH%;C:\pypy\;C:\pypy3\

Or just cd C:\pypy to test it out.

3) Ensure pip is installed (pip is a weirdly named tool for installing things).
pypy.exe -m ensurepip
4) Install a dev build of pygame.
pypy.exe -m pip install pygame --pre

5) Check if pygame is installed properly by running an included example.
pypy.exe -m pygame.examples.aliens

ps. use pypy3.exe for python 3 version of pypy above rather than pypy.exe(python 2) if that's what you like.

Tuesday, March 20, 2018

pygame on pypy usable

Hi,

TLDR; I'm at the pypy sprint, and working through the remaining pygame-on-pypy-cpyext issues.
https://youtu.be/WN1slc5O8os


Surprisingly to me... it's already usable. That is pygame (same one that runs on cpython), works on pypy through its C extension API. pypy has good support for the CPython API (through a recompile) now. PyPy is the python language with a fast JIT, so your code can approach C speeds. And in some cases it can be faster than C.

There was an issue with events stopping keyboard/mouse/etc from working. Lots of details in this issue describing the changes needed, so I hope other extensions encountering this will find it useful.
https://github.com/pygame/pygame/issues/419
But now that's fixed, every pygame app I tried on it has worked.

Cat sitting in the fog and snow of Switzerland

Why is this exciting?

This is exciting to me because:
  • pure python code being fast on pypy(after warmup), also mixed with the fast bits in C/asm.
  • cpyext is getting faster in pypy. There is already work and discussion towards it being faster than CPython.
  • maintaining one pygame code base is easier than maintaining several (pygame cffi/ctypes/cython, ...).
  • with one code base it should be fast on both pygame, and pypy(in time).
Here's our old pal solarwolf from early 2000s running on pypy.
https://youtu.be/WN1slc5O8os
Still lots of work to do (especially around PixelArray buffers and such). Then of course, there is the issue of binary wheels, so that  `pip install pygame`  works without needing to compile things from source.

How is the speed? (when do we use this tool? Is it fast enough?)

If your code is already quite well optimized, and not spending much time in python, you can't expect to see an improvement. However, if you are pushing boundaries in your python code, you can expect very good increases.

Some examples where you can expect it to be faster:
  • if profiling and a pygame function (like blit) isn't at the top of the slow bits.
  • collision detection (if you aren't using fancy algorithms).
  • a pure python ray caster.
  • writing a music synthesizer in python python.
Where it can be slower.
  • if you are going into C code for a lot of small operations. Like when using lots of pygame.Rect in a tight loop. This is because (currently) the cost of going from PyPy code into and out of CPython API code (like pygame) is a bit slow.
For me, I'm interested mostly in this for a physics art project which was really slow, and also for a software music synth written in pure python. Even more interesting is running pypy as a separate process for these tasks, and run the gui process with CPython.


Ray tracing 3D scenes. 

Here I run a python and pygame python ray tracer by Ian Mallett.

Ray Tracer to for realistic 3D lighting.

On PyPy - 18.6 seconds.
On Python 2.7 - 9 minutes, 28.1 seconds

That's 30x faster.  Making many more things possible in python - and at speed.


The fog has lifted at the Leysin pypy sprint.

Thursday, March 15, 2018

Drawing sound (as a waveform in pygame).

I recently gave an example of pygame sound generation examples, and a few people asked for more. So... here we go!

There's an infinite number of ways to visualize sound. A classic way is to draw it as a waveform.
Sound samples drawn as a Waveform. Scaled into a 320x200 sized Surface.

A sound could be made up of 44100 samples per second. Where each sample is often a 16 bit number (or 8bit or a 32bit floating point).
Python comes with a built in array for efficiently storing numbers. We can store samples in there, with integers between -32768 and 32768. This is a signed 16bit(2 byte) number. Two to the power of 16 is 65536, and if we divide that by two we see the minimum and maximum value that can hold. pow(2, 16) / 2 == 32768.

Below is the annotated code, also with an example of an array of samples representing a square wave.
We loop over the samples and draw a line from the previous sample to the current one. 0 to 1, 1 to 2, 2 to 3, ... N-1 to N.


You can also find it in the pygame CookBook at https://www.pygame.org/wiki/DrawWaveform
(maybe easier to read than on this blog)


# python built in array type is useful for storing sound data.
import array
# a square wave tone, with sample values between -32766 and 32766.
samples = array.array('h', [32766, 32766, 32766, 32766, 32766, 32766,
32766, 32766, 32766, 32766, 32766, 32766, 32766, 32766, 32766,
32766, 32766, 32766, 32766, 32766, 32766, 32766, 32766, 32766,
32766, 32766, 32766, 32766, -32766, -32766, -32766, -32766,
-32766, -32766, -32766, -32766, -32766, -32766, -32766, -32766,
-32766, -32766, -32766, -32766, -32766, -32766, -32766, -32766,
-32766, -32766, -32766, -32766, -32766, -32766, -32766])


def scale_samples_to_surf(width, height, samples):
    """ Returns a generator containing (x, y) to draw a waveform.

    :param width: width of surface to scale points to.
    :param height: height of surface to scale points to.
    :param samples: an array of signed 1 byte or signed 2 byte.
    """
    assert samples.typecode in ['h', 'b']
    # precalculate a bunch of variables, so not done in the loop.
    len_samples = len(samples)
    width_per_sample = width / len_samples
    height_1 = height - 1

    if samples.typecode == 'h':
        # for array typecode 'h', -32768 to 32768
        factor = 1.0 / 65532
        normalize_modifier = int(65532 / 2)
    elif samples.typecode == 'b':
        # for array typecode 'b', -127 to 127
        factor = 1.0 / 256
        normalize_modifier = int(256 / 2)

    return ((
        int((sample_number + 1) * width_per_sample),
        int(
            (1.0 -
                (factor *
                    (samples[sample_number] + normalize_modifier)))
            * (height_1)
        ))
    for sample_number in range(len_samples))

def draw_wave(surf,
              samples,
              wave_color = (0, 0, 0),
              background_color = pg.Color('white')):
    """Draw array of sound samples as waveform into the 'surf'.

    :param surf: Surface we want to draw the wave form onto.
    :param samples: an array of signed 1 byte or signed 2 byte.
    :param wave_color: color to draw the wave form.
    :param background_color: to fill the 'surf' with.
    """
    assert samples.typecode in ['h', 'b']
    if background_color is not None:
        surf.fill(background_color)
    width, height = surf.get_size()
    points = tuple(scale_samples_to_surf(width, height, samples))
    pg.draw.lines(surf, wave_color, False, points)

# Here we should how to draw it onto a screen.
waveform = pg.Surface((320, 200)).convert_alpha()
draw_wave(waveform, samples)
screen.fill((255, 255, 255))
screen.blit(waveform, (160, 100))
 
 

Wednesday, March 14, 2018

Sound generation pygame examples.


Here's a few sound generation examples with pygame (and no numpy/scipy).

If there's interest I'll expand this into a bigger example? Let me know.


All the basics for making a music program (sampler/synth).
  • some sample rate conversion,
  • bit rate conversion
  • tone generation using generators (square wave)
  • python arrays used as buffers for pygame.Sound (zero copy).



""" Some examples for generating and converting sounds for pygame.

Python 2.7, 3.6

Shows:
    - a simple 'square wave' generated
    - resampling sample rates (eg, 8363 to 44100)
    - using built in python array for making pygame.Sound samples.
    - samples at different bit sizes
    - converting from signed 8 to signed 16bit
    - how initializing the mixer changes what samples Sound needs.
    - Using the python stdlib audioop.ratecv for sample rate conversion.

Square Wave
  https://en.wikipedia.org/wiki/Square_wave
MOD (file format)
  https://en.wikipedia.org/wiki/MOD_(file_format)

pygame.mixer.get_init
    https://www.pygame.org/docs/ref/mixer.html#pygame.mixer.get_init
pygame.Sound
    https://www.pygame.org/docs/ref/mixer.html#pygame.mixer.Sound

array (python stdlib)
    https://docs.python.org/3/library/array.html
wave (python stdlib)
    https://docs.python.org/3/library/wave.html
audioop.ratecv (python stdlib)
    https://docs.python.org/3/library/audioop.html?highlight=audio#audioop.ratecv

"""

from array import array
import pygame as pg

class Tone(pg.mixer.Sound):
    """This generates a 'Square wave' with a generator.

    Then creates an array of samples, and passes that into pygame.Sound.
    """

    def __init__(self, frequency, array_type, volume=.1):
        self.frequency = frequency
        if array_type == 'b':
            # we have to convert the 1 byte 'b' samples to 2 byte 'h'.
            samples = self.signed_char_to_signed_short(
                self.make_samples_b()
            )
        elif array_type == 'h':
            samples = self.make_samples_h()
        else:
            raise ValueError('array_type not supported')

        pg.mixer.Sound.__init__(self, buffer=samples)
        self.set_volume(volume)

    def make_samples_b(self):
        """ Builds samples array between -127 and 127.
            Array type 'h'.
        """
        mixer_frequency = pg.mixer.get_init()[0]
        mixer_format = pg.mixer.get_init()[1]
        period = int(round(mixer_frequency / self.frequency))
        max_amplitude = 2 ** (abs(mixer_format) - 1) - 1
        max_amplitude = int(max_amplitude / 256)
        # print(f'mixer_frequency:{mixer_frequency}, mixer_format:{mixer_format}')
        # print(f'period:{period}, max_amplitude:{max_amplitude}')

        # 'b' array is signed char, 1 byte
        # https://docs.python.org/3/library/array.html
        samples = array('b',
            (max_amplitude if time < period / 2 else -max_amplitude
                for time in range(period))
        )
        return samples

    def signed_char_to_signed_short(self, b_samples):
        """ Converts 1 byte signed char samples to 2 byte signed short.

            127 to 32767
        """
        # just a simple linear conversion.
        factor = int(32767 / 127)
        return array('h', (sample * factor for sample in b_samples))

    def make_samples_h(self):
        """ Builds samples array between -32767 snd 32767.
            Array type 'h'.
        """
        mixer_frequency = pg.mixer.get_init()[0]
        mixer_format = pg.mixer.get_init()[1]
        period = int(round(mixer_frequency / self.frequency))
        max_amplitude = 2 ** (abs(mixer_format) - 1) - 1
        # print(f'mixer_frequency:{mixer_frequency}, mixer_format:{mixer_format}')
        # print(f'period:{period}, max_amplitude:{max_amplitude}')

        # 'h' array is signed short, 2 bytes
        # https://docs.python.org/3/library/array.html
        samples = array('h',
            (max_amplitude if time < period / 2 else -max_amplitude
                for time in range(period))
        )
        return samples


class Sample(pg.mixer.Sound):
    """ For playing a sample.

    Takes a file, and reads it in as 8bit signed data.

    Then converts it to the 16bit signed size the pygame.mixer needs.
    """
    def __init__(self, fname, volume=.1):
        with open(fname, 'rb') as f:
            samples = self.signed_char_to_signed_short (
                array('b', f.read())
            )
            pg.mixer.Sound.__init__(self, buffer=samples)
        self.set_volume(volume)

    def signed_char_to_signed_short(self, b_samples):
        """ Converts 1 byte signed char samples to 2 byte signed short.
            127 to 32767
        """
        # just a simple linear conversion.
        import time
        t0=time.time()
        factor = int(32767 / 127)
        samples = array('h', (
            max(sample, -127) * factor if sample < 0 else
            min(sample, 127) * factor
            for sample in b_samples))
        t1=time.time()
        print(t1-t0)
        return samples


def fetch_example_mod_file(mod_fname):
    """ Grab a file that has a sound samples in it from the net.

    'MOD is a computer file format used primarily to represent music,
    and was the first module file format. MOD files use the ".MOD"
    file extension, except on the Amiga which doesn't rely on
    filename extensions, instead it reads a file's header to
    determine filetype. A MOD file contains a set of instruments in
    the form of samples, a number of patterns indicating how and when
    the samples are to be played, and a list of what
    patterns to play in what order.'

    https://en.wikipedia.org/wiki/MOD_(file_format)
    """
    import os
    url = 'https://api.modarchive.org/downloads.php?moduleid=101996'

    if not os.path.exists(mod_fname):
        import urllib2
        print ('Fetching %s .mod into file: %s' % (url, mod_fname))
        data = urllib2.urlopen(url).read()
        with open(mod_fname, 'w') as modf:
            modf.write(data)


def resample(mod_fname):
    """ An example of resampling audio to a different framerate.

    eg, from 8363 one byte samples per second to
        44100 two byte samples per second.
    """
    import audioop
    import wave
    from io import BytesIO

    in_framerate = 8363
    in_sampwidth = 1
    in_nchannels = 1

    out_framerate = 44100

    num_seconds = 5
    with open(mod_fname, 'rb') as f:
        # Throw away the start data of this mod file.
        #   Better samples later on.
        f.read(8363*2)
        in_frame_data = f.read(in_framerate * num_seconds)

    # https://docs.python.org/3/library/audioop.html?highlight=audio#audioop.ratecv
    newfragment, newstate = audioop.ratecv(
        in_frame_data,
        in_sampwidth,
        in_nchannels,
        in_framerate,
        out_framerate,
        None)

    # print(f'len(newfragment):{len(newfragment)}')
    # A perfect conversion is not possible, because the sample
    #   rates do not divide equally. However, the number
    #   of samples should be close.
    assert (out_framerate * num_seconds) - len(newfragment) < 10
    pg.mixer.Sound(buffer=newfragment).play(-1)



# TODO:
# Converting between modo and stereo?
#   audioop.tomono and audioop.tostereo
#   https://docs.python.org/3/library/audioop.html?highlight=audio#audioop.tomono

# How to draw a wave form?
#   using pygame.draw.lines transforming audio into
#   Surface space.
#   Meaning, scaling audio samples into a particular
#   sized part of the screen.

# More sound generator types.
#   Saw tooth.



if __name__ == "__main__":
    # https://www.pygame.org/docs/ref/mixer.html#pygame.mixer.init
    pg.mixer.pre_init(44100, -16, 1, 1024)
    pg.init()

    pg.display.set_caption('Playing square wave, 808 frequency')
    pg.display.set_mode((320, 200))

    mod_fname = 'outrun_3.mod'

    fetch_example_mod_file(mod_fname)

    # play on repeat, -1 means loop indefinitely.
    # https://www.pygame.org/docs/ref/mixer.html#pygame.mixer.Sound.play
    if 0:
        Tone(frequency=808, array_type='b').play(-1)

    if 0:
        try:
            Sample(mod_fname).play(-1)
        except IOError:
            print ('no %s' % mod_fname)
    if 0:
        pg.mixer.music.load(mod_fname)
        pg.mixer.music.play()

    if 1:
        resample(mod_fname)

    going = True
    while going:
        for e in pg.event.get():
            if e.type in [pg.QUIT, pg.KEYDOWN]:
                going = False

Tuesday, March 13, 2018

pypy sprint

Been doing a bit of reading this morning on pypy things.

Trying to prepare myself a bit for the coming 'winter sprint'. One topic I'm interested in for the sprint is finishing off remaining issues for pygame running on pypy. Probably not the most common reason for people to go to the Swiss mountains I guess! So this means learning about the cpyext part of pypy... the one which allows it to use the python C API, and modules that use it.

Open source sprints are great. A bunch of people get together trying to make things. A very rare time when people get to meet other people-from-the-internet. It's a great place to learn, and to see how projects you're interested in are made.

I guess I'm a little bit too excited, and started playing with pypy early. Below are some notes from my investigations finding out how to do things.

Getting started, finding docs.

There's a getting started guide for pypy which has some basic notes on running tests, and compiling stuff.

Also, cython has a pypy section in their user guide - http://cython.readthedocs.io/en/latest/src/userguide/pypy.html

cpyext is the module which implements the python C API in pypy.

A wiki page called 'c-api' contains some documentation including a talk on cpyext. It's a little bit dated (from 7 years ago), but still interesting reading(and watching).

This wiki page was interesting:  "In which we document cpyext compatibility progress, this time with cython and pandas."
https://bitbucket.org/pypy/pypy/wiki/cpyext_2_-_cython%20and%20pandas

It's a blog of development of pypy cpyext getting numpy and pandas features working. From this I noticed a few interesting techniques... like running tests separated processes. Very useful when debugging. There is also an older document called 'Adventures in cpyext compatibility (ended)'. With even more notes on issues that came up, along with their work arounds.

Of course the Python C API docs are also useful when one wants to do Python C API things.

Use the source (Luke).

But the best source of information on cpyext seems to be the source code itself.
https://bitbucket.org/pypy/pypy/src/default/pypy/module/cpyext/

Lots and lots and lots of tests. And lots and lots of code. Bed time reading perhaps.

It seems to be a whole implementation of CPython, with bridges into pypy. But with tests. Lots and lots of tests for the python C API.

What are useful debugging techniques?

CPython has a thing called a debug build which is useful.
https://docs.python.org/3/c-api/intro.html#debugging-builds

I don't know of any special debug tools for pypy (gdb extensions or such like).


Wheels on pypy?

I found this page "pypy binary wheels for some popular packages", and a blog post about it.

It seems pypy doesn't compile on the ancient centos 5 that many linux uses. Perhaps the new manylinux which uses a newer linux will work.

It seems the piwheels project (compiles wheels for raspberry pi automatically) doesn't have pypy support yet. piwheels is really great, in that it makes lots of ARM builds available so people don't need to compile stuff themselves on the raspberrypi.

No idea if windows, or mac binary wheels work for pypy. Without binary wheels for these platforms, most pypy users won't be able to install a C based library like pygame on pypy.

Which platform?

It seems linux amd64 is the best supported, but I'm also interested in Mac, and Windows builds.
Especially setting them up on Appveyor, and Travis /Mac.

Translating on windows is the document which describes the windows builds. Not sure my cheap windows laptop has enough memory for it. It's only got 4GB of ram, and I expect it's going to take hours to compile.

The buildbot contains Nightly builds for windows as well as other platforms. So I might try using them to debug stuff, and I guess do extension development. Failing that, maybe I can copy a build off someones computer!

Hopefully I won't need to hire some remote servers for compiling stuff. It doesn't seem like I'd need to be recompiling all of pypy each time for extension development.

Thursday, March 08, 2018

pygameweb - 0.0.4 - Northern Rockhopper Penguin

pygameweb is the source code for the pygame website at https://www.pygame.org/


Northern Rockhopper Penguin (photo by Brian Gratwicke)
  • #52 Adding ability to deploy through travisci. And docs.
  • #46 Wiki readability. Wiki tools on left out of reading line.
  • Twitter card for releases. For nicer view when posting to twitter.
  • Wiki table of contents improvements.
  • builds docs for pygame, when something lands on pygame github master branch.
  • project release feeds bugfix.
  • Only show one recent release for each project. Some projects release a few times in a row.
  • Wiki, nicer pre code handling.
  • Wiki code uses inline colors, so copy/paste works with colors too.
  • use https for login and register buttons, even when on http (see Why can't we turn off HTTPS?).
  • Ask more bad robot web crawlers not to be bad.
See a full diff of changes since last release.

Tuesday, March 06, 2018

On the threshold of a journey with a Jedi, a Sphinx, and a Homebrew.

Feels like practice.
It all started some days ago, late at night,
with a comment made by someone on the internet...
"Threshold does not take keyword arguments"
As is often the case when you want to correct
something on the internet,
this comment lead me on a journey
(not one where I sell numpy for 22 million monies,
a different type of journey
involving text editors and compilers,
and lots of failing tests).
Software archeology -- not with Indiana,
but at least there was a Jedi and a Sphinx.

"Relax. Don't worry. And have a homebrew." -- Charlie Papazian

And with a warm brew in hand,
herein begins my tale...

Starting at the end, where all poor stories start,
with the comment I will send "TheBob427" (actual real nick),
right after I publish this blog post... [-1]





Dear TheBob427,

yes, this function doesn't take keyword arguments and you are right
that it's super buggy. However, now after some days of bashing my keyboard,
it should actually work. Additionally it should be slightly less weird.
It was a fun journey. I guess you'll have to wait for the next release to use it.
Which should be here 'soon'. Which in open source minutes is 
approximately the same amount of time as 'when it's done'.

yours sincerely,
illume

ps. you could also use PixelArray.replace instead.



So, I updated the pygame.transform.threshold function, tests and docs [0].
Apart from that function being buggy and super confusing...

It was sort of a test of how we could improve all the pygame functions for pygame 3000.
Whilst it still doesn't follow the new pygame C API yet[1],
I wanted to try a few things like 'new' C99/C11 features
(thanks ffmpeg for tricking MS into dragging their compiler into the 90s),
new docs features (param types, and examples) and such like.

It's quite a bit of C and python test changes,
and took me quite a few days longer than I expected.
Simplifying here. Simplifying there. Simplifying everywhere.

One thing it does now is use sphinx type parameters.
Apart from the docs being improved, tools like
jedi can use them for autocomplete and type inference.

So, now editors like vim, sublime, and vscode can autocomplete with pygame.
(I had to change a few things for the 'odd' but 'cool' pygame
'missing module' imports feature so the type inference worked in jedi [2])

Fancy.

For examples, sphinx can now easily include source code.
So we can include pygame.examples code, or even test cases.
The 'nulledge' python search code search doesn't seem all that good
for many pygame things, and it lacks https. So i'm thinking of dropping
that and manually linking to relevant examples/test code.

Additionally, we can link to github source code, and even
the documentation files themselves. Which is handy for
'edit this'/'fork on github' style buttons.


Feels like practice.



[-1] a comment, https://www.reddit.com/r/pygame/comments/80zyqg/threshold_does_not_take_keyword_arguments/dvae53d/
[0] threshold function pr, https://github.com/pygame/pygame/pull/408
[1] pygame C api issue, https://github.com/pygame/pygame/issues/333
[2] jedi confused with pygame pr, https://github.com/pygame/pygame/pull/413

Thursday, March 01, 2018

Pre release pygame builds through pypi and pip.

(If there's any python packaging experts reading this, please comment if this doesn't seem reasonable. https://github.com/pygame/pygame/issues/409 You will get 42 internet points, and a cookie if you ever visit Berlin).

We want to be able to more widely test pre release builds, to avoid bugs and try things out on different OS/hardware combinations. This is the plan on how it will be done for pygame.

---

Pre release 'dev' and release candidate('rc') builds of pygame through pip and pypi.
  • every merge into master releases a dev build. 1.9.4.dev1+git.77.facebeef123
  • every release candidate tagged releases an rc build. 1.9.4.rc1

How to install pre release builds.

Pip pre-release documentation
`pip install pygame --pre`
Requiring a specific development build is possible according to the documentation.
`pip install pygame==1.9.4.dev1`

Version naming.

From packaging guide pre release versioning:
    zero or more dev releases (denoted with a “.devN” suffix)
    zero or more alpha releases (denoted with a “.aN” suffix)
    zero or more beta releases (denoted with a “.bN” suffix)
    zero or more release candidates (denoted with a “.rcN” suffix)
  • Dev versions will be named like this: 1.9.4.dev1+git.77.facebeef123 (version+local_version). With the dev1 becoming dev2, dev3. git describe can be used to get this info.
  • pygame.version.rev should have the git hash https://www.pygame.org/docs/ref/pygame.html#pygame.version.rev
  • Uploading wheels currently only happens when a tag happens. (eg, 1.9.3). Dev builds would instead need to happen when something is merged into master.
  • release candidates can be done with a tag. 1.9.4.rc1 will be release candidate 1.

git describe, for getting commits since last release.

Can be used to get the number of commits since the last release.
git describe --tags
1.9.3-124-g2d842abc
Using this we can grab the info we need to build the pypi-happy version strings.
Note, it needs to use --tags if we don't have annotated tags. (which we should use going forwards).

Dev builds to test pypi instead?

No.
  • piwheels don't download from testpypi, so we need to release to pypi to get these.
  • Also, the packaging guide doesn't talk about doing this. (it's just used for testing, and the DB is wiped occasionally).
  • Will pypi be cluttered with lots of dev releases? 3-20 per month (* 30 files)? I don't think this is a problem. There's nothing in the packaging guide to say don't do this.
  • people may have a need to rely on a specific dev version (especially since pygame is released infrequently), and testpypi gets wiped.

Sunday, February 25, 2018

Why can't we turn off HTTP?

Currently, I'm sitting in a library not able to access HTTPS (probably... I guess it depends when you're reading this) for a particular website.

I like to spend time in libraries. Especially libraries in exotic far off places. Surrounded by book-smell, and curious people learning. Unfortunately some libraries have shockingly bad internet. Some of them decide they will filter which websites people can look at so that they are safe from viruses and nasty stuff which damages their computers.



It's exciting that 60-70% of internet connections to some websites are encrypted these days. But why not turn HTTP off and force everyone to use HTTPS?
Indeed. Why not turn off HTTP completely? I'll try to answer that here (for pygame.org; your website may be different).



Unfortunately these filters in libraries are sometimes pretty terrible and ancient. And when you travel, and spend time in libraries and use hotel internet you begin to notice that the internet that you know at home or work is kind of not the same all around the world. A shocking revelation I know things are different in different places. Soon there will be research funded to confirm this for you in ten years.

One thing they do is have proxy servers that jump in the middle between the browser and the server. Sometimes in strange ways that don't quite work.



Oh. So this is why I can't internet-all-the-things whilst sitting in a hammock on a small island.



Universities are another type of place I've had bespoke-internet-experiences.

So, this is one reason I like to still offer things over normal HTTP, because some people are blocked from HTTPS. A person can still choose already to use HTTPS (which technically still does have a MITM attacks going on from university/corp/country proxies and the like). We could also use HSTS (https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Strict-Transport-Security) which is the proper solution rather than redirects, but that means HTTP doesn't work. Because leaving HTTP on means it's possible HTTPS can be worked around.

If you're privileged enough to have good internet then, please take our EFF friends advice and use the https-everywhere addon for your browser. Unfortunately some things will still break for you. https://www.eff.org/https-everywhere

However, could we still do better? And still make it accessible for those people who are stuck with horrible internet connections. Yes, of course!

So, now pygame.org URLs permanently redirect to https://www.pygame.org/, and the login URLs even on non-HTTPS point to HTTPS. Additionally, I've updated lots of links in the docs and on the internet to HTTPS URLs, and will continue doing that where possible. Search engines always link to HTTPS (if available) it seems, and adding canonical HTTPS link tags in the headers is being tracked here: #34



I would like to update hundreds of mixed HTTP/HTTPS documents to turn all the HTTP links to new HTTPS links... but instead I chose life. Is there a header for that? Yes there is... pop on over to https://w3c.github.io/webappsec-upgrade-insecure-requests/ and learn all about it.



Browsers are now rolling out HTTPS upgrade headers, where they send a hint that they can go to a HTTPS URL when one is available. https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Upgrade-Insecure-Requests
You can also make a content security policy which allows the browser to upgrade old unecrypted links to HTTPS ones. You're telling the browser "Dear browser, everything should work with HTTPS, but I haven't updated all the URLs yet". Seems to be at 77% of browser support at the moment, and even the latest version of Edge supports it. https://caniuse.com/#feat=upgradeinsecurerequests

But remember, even if a browser supports it, it doesn't mean it hasn't been configured to use a proxy in the middle which has issues with HTTPS. Or that 500ms latency turns those several extra back-and-forths HTTPS requires into an unbearable thing to use. With three 500ms round trips, that turns into more than 1500ms. Compared to 20ms*3==60ms for someone on a fast connection nearby.

The reason we can't have zero round trips with HTTPS everywhere I hear you ask? That should would be good for people in far off lands with bad internet connections. Let me tell you my friend those same proxies, and Idiot of Things with ancient buggy HTTPS stacks. Blame your old printer.
 
But I've enabled 0-rtt and HTTP/2 anyway, and the reason is that those same broken internet connects are going to be broken anyway. Might as well make it slightly faster and safer for everyone else, and the broken connections can use HTTP which they would have to do anyway.



Is TLS fast Yet? Yes. Sort of. https://istlsfastyet.com/



With these changes, there's been a 30%(handwavy-made-up-stat) increase in HTTPS connections, and now the majority of connections are using HTTPS. Hopefully as more links are updated (18 years of links on the interwebs might not ever be upgraded).
Cloudflare (which pygame.org is using for HTTPS) supports HTTPS on some pretty ancient computers (we are paying for the pro plan). Even my android 2 phone works with it. https://support.cloudflare.com/hc/en-us/articles/203041594-What-browsers-work-with-Cloudflare-s-SSL-certificates-
 
www.pygame.org is still open for business fun on HTTP however, even though most people probably won't go there 'soon'.  The people in far off places with weird internet connections, and those retro computing people with the browser user agent string "AmigaVoyager/3.2 (AmigaOS/MC680x0)" will still be able to download their files.

pygame.org is //retro computer friendly// after all. But we're also friendly to people in far off lands sitting in hammocks and to people who like book-smells + learning.

Thanks for reading.