I tried adding post-increment to CPython. Overview and summary

I tried adding post-increment to CPython

Implemented increment in CPython. This time, I will introduce the outline of increment and the knowledge obtained. In the next implementation edition, we will look at the implementation method in chronological order.

Link list

All three times plus extra edition. Overview and summary of adding post-increment to CPython Implementation of CPython with post-increment List of all changes when adding post-increment to CPython Extra edition of adding post-increment to CPython

slide

First of all, please have a brief summary of the results slide.

specification

It works like this.


>>> i=0
>>> i++
0
>>> i
1


>>> lst=[x for x in range(5)]
>>> lst
[0, 1, 2, 3, 4]
>>> lst[0]++
0
>>> lst
[1, 1, 2, 3, 4]

>>> class cls:
...     a=5
...
>>> cls_obj=cls()
>>> cls_obj.a
5
>>> cls_obj.a++
5
>>> cls_obj.a
6

Increment can be implemented not only for variables but also for lists and member variables. In addition, the evaluation is returned at the same time as the variable is rewritten.

Obtained findings

Here's what I found by looking at the CPython 3.5.0 source code when implementing increments.

About the behavior of CPython

The Python script is executed as follows.

--Lexical analysis - Include/tokenizer.h - Parser/tokenizer.c

How to change Python grammar

There is a page called 23. Changing CPython ’s Grammar in Python Developer's Guide, so you can refer to it. However, this is not enough words, so I've summarized some more specific changes.

  1. When you want to change the reserved word to another word such as "I want to write foreach instead of for" --Sometimes you just need to change Grammar / Grammar appropriately --You only need to change the 'for' part to " ('for' |'foreach') , but if you rewrite the deep'elif', it will be in Python / ast.c. It seems that it is necessary to change around ast.c because it will fall with the assert of (unconfirmed)

  2. If you want to use a symbol string that is not used in Python, such as "I want to be able to use! Instead of for" --In addition to modifying Grammar / Grammar as above, define tokens in Include / tokenizer.h, Parser / tokenizer.c

  3. If you want to add a grammar that uses symbols that are already used in other meanings -For example, list comprehension[x + 1 | x <- range(10)]Change to be able to write (|Is already used in the sense of bit OR) -Since the carved tokens are passed to the automatically generated parser in sequence, the bit operation is performed at the timing before that.|Or list comprehension|I thought it would be good to judge whether it was, but I gave up because it would take time to make this from scratch. ――I think it should be done by the parser in the first place, but is the Python grammar, which is originally LL (1), no longer LL (1)? → It seems necessary to read the automatic generation part of the parser that I did not touch and know the ability of the parser --I didn't quite understand (for that reason, in this experiment, I implemented using $ instead of | and cheated)

  4. If you want to define some syntactic sugar --In addition to tokenizer. *, Grammar, it seems good to make it into a synonymous tree with Python / ast.c

  5. If you want to add a grammar that goes beyond the framework of expressions and sentences or does not exist in existing ones --tokenizer, Grammar, .. Sometimes compile.c should also spit out nice bytecode. If necessary, also define new opcodes and their interpretations

By the way, if you make a destructive change, you will not be able to compile the library at the time of make install.

In this increment implementation, there was at least one expression ++ 2 and an error was thrown (it is a mystery why it is written like that in the first place).


compound_stmt: if_stmt | while_stmt | for_stmt | try_stmt | with_stmt | funcdef | classdef | decorated
if_stmt: 'if' test ':' suite ('elif' test ':' suite)* ['else' ':' suite]
while_stmt: 'while' test ':' suite ['else' ':' suite]
for_stmt: 'for' exprlist 'in' testlist ':' suite ['else' ':' suite]

Recommended Posts

I tried adding post-increment to CPython. Overview and summary
I tried adding post-increment to CPython Implementation
I tried adding post-increment to CPython Extra edition
I tried adding post-increment to CPython. List of all changes
I tried adding system calls and scheduler to Linux
I tried using PyEZ and JSNAPy. Part 1: Overview
I implemented DCGAN and tried to generate apples
I tried adding VPS to ConoHa ~ SSH connection
[Introduction to PID] I tried to control and play ♬
I tried to debug.
I tried to paste
I tried to read and save automatically with VOICEROID2 2
I tried to implement and learn DCGAN with PyTorch
I tried to automatically read and save with VOICEROID2
I tried to implement Grad-CAM with keras and tensorflow
I tried to install scrapy on Anaconda and couldn't
I tried to learn PredNet
I tried to organize SVM.
I tried to implement PCANet
I tried to reintroduce Linux
I tried to introduce Pylint
I tried to summarize SparseMatrix
I tried to touch jupyter
I tried to implement StarGAN (1)
I tried to predict and submit Titanic survivors with Kaggle
I tried to combine Discord Bot and face recognition-for LT-
I tried to get Web information using "Requests" and "lxml"
I tried to illustrate the time and time in C language
I tried to display the time and today's weather w
[Introduction to infectious disease model] I tried fitting and playing ♬
I tried to enumerate the differences between java and python
I tried to make GUI tic-tac-toe with Python and Tkinter
I tried to implement Deep VQE
I tried to touch Python (installation)
I tried to visualize bookmarks flying to Slack with Doc2Vec and PCA
I tried to implement adversarial validation
I tried to let Pepper talk about event information and member information
I tried to explain Pytorch dataset
I tried Watson Speech to Text
I tried to make a periodical process with Selenium and Python
I tried to touch Tesla's API
I tried to create Bulls and Cows with a shell program
I tried to implement hierarchical clustering
I tried to organize about MCMC.
I tried to easily detect facial landmarks with python and dlib
I tried to extract players and skill names from sports articles
I tried to implement Realness GAN
I tried to move the ball
I tried to estimate the interval.
I tried moving the image to the specified folder by right-clicking and left-clicking
I tried to visualize the age group and rate distribution of Atcoder
I tried to express sadness and joy with the stable marriage problem.
[Deep Learning from scratch] I tried to implement sigmoid layer and Relu layer.
I tried to convert datetime <-> string with tzinfo using strftime () and strptime ()
I tried to learn the angle from sin and cos with chainer
I tried to make Kana's handwriting recognition Part 2/3 Data creation and learning
I tried to extract and illustrate the stage of the story using COTOHA
I tried to verify and analyze the acceleration of Python by Cython
[Markov chain] I tried to read quotes and negative emotions into Python.
I tried to classify Oba Hana and Emiri Otani by deep learning
I tried to create a sample to access Salesforce using Python and Bottle